<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="/default.xsl"?>
<fr:tree xmlns:fr="http://www.forester-notes.org" xmlns:html="http://www.w3.org/1999/xhtml" xmlns:xml="http://www.w3.org/XML/1998/namespace" root="false" base-url="/">
  <fr:frontmatter>
    <fr:authors>
      <fr:author>
        <fr:link href="https://shelter.sirref.org/Shelter Programmers/" type="external">Shelter Programmers</fr:link>
      </fr:author>
    </fr:authors>
    <fr:date>
      <fr:year>2025</fr:year>
      <fr:month>11</fr:month>
      <fr:day>17</fr:day>
    </fr:date>
    <fr:uri>https://shelter.sirref.org/logs/</fr:uri>
    <fr:display-uri>logs</fr:display-uri>
    <fr:route>/logs/</fr:route>
    <fr:title text="Shelter Logs">Shelter Logs</fr:title>
  </fr:frontmatter>
  <fr:mainmatter>
    <html:p>Weekly, or there abouts, updates on the development of <fr:link href="/index/" title="Shelter" uri="https://shelter.sirref.org/index/" display-uri="index" type="local">Shelter</fr:link>.</html:p>
    <fr:tree show-metadata="false" expanded="false" toc="false" numbered="false">
      <fr:frontmatter>
        <fr:authors>
          <fr:author>
            <fr:link href="https://shelter.sirref.org/Patrick Ferris/" type="external">Patrick Ferris</fr:link>
          </fr:author>
        </fr:authors>
        <fr:date>
          <fr:year>2025</fr:year>
          <fr:month>12</fr:month>
          <fr:day>9</fr:day>
        </fr:date>
        <fr:uri>https://shelter.sirref.org/shelter-log-002/</fr:uri>
        <fr:display-uri>shelter-log-002</fr:display-uri>
        <fr:route>/shelter-log-002/</fr:route>
        <fr:title text="@ How?">@ How?</fr:title>
      </fr:frontmatter>
      <fr:mainmatter>
        <html:p>A quick update on a new feature that was added recently, the <html:code>@ how</html:code> <fr:link href="/shelter-0002/" title="Shelter Meta-commands" uri="https://shelter.sirref.org/shelter-0002/" display-uri="shelter-0002" type="local">meta-command</fr:link>.</html:p>
        <html:p><![CDATA[One of the issues facing any exploratory programmer (or indeed anybody]]>  <![CDATA[returning to a project) is the]]> <html:em>provenance</html:em> of a given file. How that file came to be and who and what processes have interacted with it since.</html:p>
        <html:p>Shelter now has a command for asking exactly that of any given file in the system.</html:p>
        <html:pre><![CDATA[echo hello > hello.txt
echo world > world.txt
cat world.txt > hello.txt
@ how hello.txt]]></html:pre>
        <html:p>The previous snippet of shell script will output:</html:p>
        <html:pre><![CDATA[Ran echo hello > hello.txt
Ran cat world.txt > hello.txt (read: "world.txt")]]></html:pre>
        <html:p>There are plenty of optimisations and improvements to make, but this is a nice first step to showing the kinds of feature you get when programming with Shelter.</html:p>
      </fr:mainmatter>
    </fr:tree>
    <fr:tree show-metadata="false" expanded="false" toc="false" numbered="false">
      <fr:frontmatter>
        <fr:authors>
          <fr:author>
            <fr:link href="https://shelter.sirref.org/Shelter Programmers/" type="external">Shelter Programmers</fr:link>
          </fr:author>
        </fr:authors>
        <fr:date>
          <fr:year>2025</fr:year>
          <fr:month>11</fr:month>
          <fr:day>16</fr:day>
        </fr:date>
        <fr:uri>https://shelter.sirref.org/shelter-log-001/</fr:uri>
        <fr:display-uri>shelter-log-001</fr:display-uri>
        <fr:route>/shelter-log-001/</fr:route>
        <fr:title text="Import statements">Import statements</fr:title>
      </fr:frontmatter>
      <fr:mainmatter>
        <html:p>Welcome to the first <fr:link href="/index/" title="Shelter" uri="https://shelter.sirref.org/index/" display-uri="index" type="local">Shelter</fr:link> log! In this log I will explain the work I have been doing to add an <html:code>@ import</html:code> <fr:link href="/shelter-0002/" title="Shelter Meta-commands" uri="https://shelter.sirref.org/shelter-0002/" display-uri="shelter-0002" type="local">meta-command</fr:link>.</html:p>
        <html:p>The syntax thus far is relatively straightforward.</html:p>
        <html:pre><![CDATA[@ import --name=<optional-name> <URI> <DST>]]></html:pre>
        <html:p>There are a few key principles underlying <html:code>@ import</html:code>.</html:p>
        <html:ol>
          <html:li>
            <html:p>It should be at least as expressive as Docker's <html:code>COPY ...</html:code> command.</html:p>
          </html:li>
          <html:li>
            <html:p>It should deal with a wider variety of import sources. In addition to local <![CDATA[file paths this could be data over HTTP (]]> <html:code>https://</html:code><![CDATA[), git repositories, zip]]>  archives over ssh etc.</html:p>
          </html:li>
          <html:li>
            <html:p>Imports, where possible, should be catalogued and shared across <fr:link href="/at-session/" title="Shelter Meta-commands › @ session " uri="https://shelter.sirref.org/at-session/" display-uri="at-session" type="local">sessions</fr:link>.</html:p>
          </html:li>
        </html:ol>
        <fr:tree show-metadata="false">
          <fr:frontmatter>
            <fr:authors>
              <fr:author>
                <fr:link href="https://shelter.sirref.org/Shelter Programmers/" type="external">Shelter Programmers</fr:link>
              </fr:author>
            </fr:authors>
            <fr:date>
              <fr:year>2025</fr:year>
              <fr:month>11</fr:month>
              <fr:day>16</fr:day>
            </fr:date>
            <fr:title text="URIs as Sources">URIs as Sources</fr:title>
          </fr:frontmatter>
          <fr:mainmatter>
            <html:p><![CDATA[Point (1) can be dealt with by point (2). Indeed, that latest development]]>  branch already allows users to import local files and directories into their session. For example:</html:p>
            <html:pre><![CDATA[@ import shelter.opam .
opam install . --deps-only --with-test]]></html:pre>
            <html:p>By allowing arbitrary URIs in the import command, we hope to force users away from downloading data via <html:code>curl</html:code> or python scripts. Lifting these side-effectful imports into <fr:link href="https://shelter.sirref.org/shelter/" type="external">Shelter</fr:link> allows us to manage the data in a much cleaner fashion.</html:p>
          </fr:mainmatter>
        </fr:tree>
        <fr:tree show-metadata="false">
          <fr:frontmatter>
            <fr:authors>
              <fr:author>
                <fr:link href="https://shelter.sirref.org/Shelter Programmers/" type="external">Shelter Programmers</fr:link>
              </fr:author>
            </fr:authors>
            <fr:date>
              <fr:year>2025</fr:year>
              <fr:month>11</fr:month>
              <fr:day>16</fr:day>
            </fr:date>
            <fr:title text="Scripts as Sources">Scripts as Sources</fr:title>
          </fr:frontmatter>
          <fr:mainmatter>
            <html:p>One idea I had was to allow users to invoke scripts as there "import source". Something like:</html:p>
            <html:pre><![CDATA[@ import -- python download_gedi_data.py]]></html:pre>
            <html:p>And using the same <fr:link href="https://shelter.sirref.org/shelter-tracing/" type="external">eBPF tracing</fr:link> we use during normal execution we can capture a fairly good idea of the tools and files needed to perform an import.</html:p>
          </fr:mainmatter>
        </fr:tree>
        <fr:tree show-metadata="false">
          <fr:frontmatter>
            <fr:authors>
              <fr:author>
                <fr:link href="https://shelter.sirref.org/Shelter Programmers/" type="external">Shelter Programmers</fr:link>
              </fr:author>
            </fr:authors>
            <fr:date>
              <fr:year>2025</fr:year>
              <fr:month>11</fr:month>
              <fr:day>16</fr:day>
            </fr:date>
            <fr:title text="Sharing Imports">Sharing Imports</fr:title>
          </fr:frontmatter>
          <fr:mainmatter>
            <html:p>One example of better data management is that <fr:link href="https://shelter.sirref.org/shelter/" type="external">Shelter</fr:link> imports can be <![CDATA[explicitly shared across sessions via the naming mechanism (also by hash).]]></html:p>
            <html:p>When a user imports some data, they can optionally name that piece of data. For example:</html:p>
            <html:pre><![CDATA[@ import
  --name=belfast-trees \
  https://www.belfastcity.gov.uk/getmedia/262a1f01-f219-4780-835e-7a833bdd1e1c/odTrees.csv \
  /home]]></html:pre>
            <html:p><fr:link href="https://shelter.sirref.org/shelter/" type="external">Shelter</fr:link> stores that away and makes a link between the name, the URI and the underlying data. A user may then import the same piece of data into a different session by simply writing:</html:p>
            <html:pre><![CDATA[@ import --name=belfast-trees /var/lib]]></html:pre>
            <html:p><![CDATA[The same data is shared (read-only) into this session. By asking for the user's]]>  intent via the naming scheme, it makes it possible to then <html:em>version</html:em> the data and have a means by which to update said data, one could imagine:</html:p>
            <html:pre><![CDATA[@ update --name=belfast-trees]]></html:pre>
            <html:p>Which will retry the original URI and see if the data has changed, or if it has not. There is an implicit versioning number that users can specify in the name should they choose to.</html:p>
          </fr:mainmatter>
        </fr:tree>
      </fr:mainmatter>
    </fr:tree>
  </fr:mainmatter>
  <fr:backmatter>
    <fr:tree show-metadata="false" hidden-when-empty="true">
      <fr:frontmatter>
        <fr:authors />
        <fr:title text="References">References</fr:title>
      </fr:frontmatter>
      <fr:mainmatter />
    </fr:tree>
    <fr:tree show-metadata="false" hidden-when-empty="true">
      <fr:frontmatter>
        <fr:authors />
        <fr:title text="Context">Context</fr:title>
      </fr:frontmatter>
      <fr:mainmatter />
    </fr:tree>
    <fr:tree show-metadata="false" hidden-when-empty="true">
      <fr:frontmatter>
        <fr:authors />
        <fr:title text="Backlinks">Backlinks</fr:title>
      </fr:frontmatter>
      <fr:mainmatter>
        <fr:tree show-metadata="true" expanded="false" toc="false" numbered="false">
          <fr:frontmatter>
            <fr:authors>
              <fr:author>
                <fr:link href="https://shelter.sirref.org/The Shelter Developers/" type="external">The Shelter Developers</fr:link>
              </fr:author>
            </fr:authors>
            <fr:date>
              <fr:year>2025</fr:year>
              <fr:month>7</fr:month>
              <fr:day>21</fr:day>
            </fr:date>
            <fr:uri>https://shelter.sirref.org/index/</fr:uri>
            <fr:display-uri>index</fr:display-uri>
            <fr:route>/index/</fr:route>
            <fr:title text="Shelter">Shelter</fr:title>
          </fr:frontmatter>
          <fr:mainmatter>
            <html:p>Welcome to Shelter's documentation.</html:p>
            <html:p>Shelter is a <html:em>metashell</html:em>. A tool for managing shell sessions in a more reproducible way.</html:p>
            <html:p>To get started with Shelter, you can read the <fr:link href="/shelter-0003/" title="Getting Started" uri="https://shelter.sirref.org/shelter-0003/" display-uri="shelter-0003" type="local">getting started</fr:link> tutorial. For a more thorough understanding of what Shelter offers and how to use it, then please follow the <fr:link href="/shelter-0004/" title="Documentation" uri="https://shelter.sirref.org/shelter-0004/" display-uri="shelter-0004" type="local">documentation</fr:link>.</html:p>
            <fr:tree show-metadata="false" expanded="false" numbered="false">
              <fr:frontmatter>
                <fr:authors />
                <fr:date>
                  <fr:year>2025</fr:year>
                  <fr:month>7</fr:month>
                  <fr:day>21</fr:day>
                </fr:date>
                <fr:uri>https://shelter.sirref.org/shelter-0001/</fr:uri>
                <fr:display-uri>shelter-0001</fr:display-uri>
                <fr:route>/shelter-0001/</fr:route>
                <fr:title text="About Shelter's Documentation">About Shelter's Documentation</fr:title>
              </fr:frontmatter>
              <fr:mainmatter>
                <html:p>Shelter's documentation is written in using <fr:link href="https://forester-notes.org/" type="external">Forester</fr:link> via <fr:link href="https://graft.sirref.org/" type="external">Graft</fr:link>.</html:p>
                <html:p>The main ethos is to divide the documentation into fairly small, standalone chunks. These can then be pieced together into documentation pages that are tightly linked.</html:p>
              </fr:mainmatter>
            </fr:tree>
            <html:p>For reference documentation, we have:</html:p>
            <html:ul>
              <html:li>
                <html:p>The <fr:link href="/shelterfile-syntax/" title="Shelterfile Syntax" uri="https://shelter.sirref.org/shelterfile-syntax/" display-uri="shelterfile-syntax" type="local">Shelterfile syntax</fr:link> -- the syntax for shelterfiles, a mishmash of Dockerfiles and shell scripts. These are useful for setting up branches for developement <![CDATA[or running (modest) pipelines.]]></html:p>
              </html:li>
              <html:li>
                <html:p>The available <fr:link href="/shelter-0002/" title="Shelter Meta-commands" uri="https://shelter.sirref.org/shelter-0002/" display-uri="shelter-0002" type="local">meta-commands</fr:link> -- these extend past shell scripts and allow users to access the more <html:em>interesting</html:em> parts of Shelter like <html:code>@ undo</html:code> or <html:code>@ merge &lt;branch&gt;</html:code> for example.</html:p>
              </html:li>
              <html:li>
                <html:p>The <fr:link href="/logs/" title="Shelter Logs" uri="https://shelter.sirref.org/logs/" display-uri="logs" type="local">development logs</fr:link> -- a set of developer logs about the activity on shelter.</html:p>
              </html:li>
            </html:ul>
          </fr:mainmatter>
        </fr:tree>
      </fr:mainmatter>
    </fr:tree>
    <fr:tree show-metadata="false" hidden-when-empty="true">
      <fr:frontmatter>
        <fr:authors />
        <fr:title text="Related">Related</fr:title>
      </fr:frontmatter>
      <fr:mainmatter>
        <fr:tree show-metadata="true" expanded="false" toc="false" numbered="false">
          <fr:frontmatter>
            <fr:authors>
              <fr:author>
                <fr:link href="https://shelter.sirref.org/The Shelter Developers/" type="external">The Shelter Developers</fr:link>
              </fr:author>
            </fr:authors>
            <fr:date>
              <fr:year>2025</fr:year>
              <fr:month>7</fr:month>
              <fr:day>21</fr:day>
            </fr:date>
            <fr:uri>https://shelter.sirref.org/index/</fr:uri>
            <fr:display-uri>index</fr:display-uri>
            <fr:route>/index/</fr:route>
            <fr:title text="Shelter">Shelter</fr:title>
          </fr:frontmatter>
          <fr:mainmatter>
            <html:p>Welcome to Shelter's documentation.</html:p>
            <html:p>Shelter is a <html:em>metashell</html:em>. A tool for managing shell sessions in a more reproducible way.</html:p>
            <html:p>To get started with Shelter, you can read the <fr:link href="/shelter-0003/" title="Getting Started" uri="https://shelter.sirref.org/shelter-0003/" display-uri="shelter-0003" type="local">getting started</fr:link> tutorial. For a more thorough understanding of what Shelter offers and how to use it, then please follow the <fr:link href="/shelter-0004/" title="Documentation" uri="https://shelter.sirref.org/shelter-0004/" display-uri="shelter-0004" type="local">documentation</fr:link>.</html:p>
            <fr:tree show-metadata="false" expanded="false" numbered="false">
              <fr:frontmatter>
                <fr:authors />
                <fr:date>
                  <fr:year>2025</fr:year>
                  <fr:month>7</fr:month>
                  <fr:day>21</fr:day>
                </fr:date>
                <fr:uri>https://shelter.sirref.org/shelter-0001/</fr:uri>
                <fr:display-uri>shelter-0001</fr:display-uri>
                <fr:route>/shelter-0001/</fr:route>
                <fr:title text="About Shelter's Documentation">About Shelter's Documentation</fr:title>
              </fr:frontmatter>
              <fr:mainmatter>
                <html:p>Shelter's documentation is written in using <fr:link href="https://forester-notes.org/" type="external">Forester</fr:link> via <fr:link href="https://graft.sirref.org/" type="external">Graft</fr:link>.</html:p>
                <html:p>The main ethos is to divide the documentation into fairly small, standalone chunks. These can then be pieced together into documentation pages that are tightly linked.</html:p>
              </fr:mainmatter>
            </fr:tree>
            <html:p>For reference documentation, we have:</html:p>
            <html:ul>
              <html:li>
                <html:p>The <fr:link href="/shelterfile-syntax/" title="Shelterfile Syntax" uri="https://shelter.sirref.org/shelterfile-syntax/" display-uri="shelterfile-syntax" type="local">Shelterfile syntax</fr:link> -- the syntax for shelterfiles, a mishmash of Dockerfiles and shell scripts. These are useful for setting up branches for developement <![CDATA[or running (modest) pipelines.]]></html:p>
              </html:li>
              <html:li>
                <html:p>The available <fr:link href="/shelter-0002/" title="Shelter Meta-commands" uri="https://shelter.sirref.org/shelter-0002/" display-uri="shelter-0002" type="local">meta-commands</fr:link> -- these extend past shell scripts and allow users to access the more <html:em>interesting</html:em> parts of Shelter like <html:code>@ undo</html:code> or <html:code>@ merge &lt;branch&gt;</html:code> for example.</html:p>
              </html:li>
              <html:li>
                <html:p>The <fr:link href="/logs/" title="Shelter Logs" uri="https://shelter.sirref.org/logs/" display-uri="logs" type="local">development logs</fr:link> -- a set of developer logs about the activity on shelter.</html:p>
              </html:li>
            </html:ul>
          </fr:mainmatter>
        </fr:tree>
      </fr:mainmatter>
    </fr:tree>
    <fr:tree show-metadata="false" hidden-when-empty="true">
      <fr:frontmatter>
        <fr:authors />
        <fr:title text="Contributions">Contributions</fr:title>
      </fr:frontmatter>
      <fr:mainmatter />
    </fr:tree>
  </fr:backmatter>
</fr:tree>
