Data GClasses

Time-series, graph database, and resource persistence.

Source: kernel/c/root-linux/src/c_tranger.c, c_treedb.c, c_node.c, c_resource2.c

C_TRANGER¶

Time-range database manager — wraps timeranger2 for CRUD operations on time-series topics.

master answers what the tranger IS. It is configured (SDF_RD) and read at create, but read afterwards it answers the tranger’s own flag: a master that lost its lock while stopped (another process took the store) is a replica, and gobj_read_bool_attr(gobj_tranger, "master") says FALSE (new after 7.25.4; 7.25.4 answered the configuration). open-rt and open-list follow it: a replica’s feed is an rt_disk, which the other master’s appends reach through the disk. The flag moves when the tranger REVIVES, which happens at the first write or the first topic open after tranger2_stop(). Between the stop and that moment it still says what it was, so an open-list on a topic that is already open can still get an rt_mem feed. Read master again after the restart has opened its topics.

Commands¶

Who gets the realtime feed. EV_TRANGER_RECORD_ADDED is published to every subscriber: a remote one keeps to its own feed with a __filter__ on its rt_id. The appends a LIVE list pushes (open-list without return_data) carry the list_id as their rt_id, the same way: __filter__: {rt_id: "pepe"} takes the pushes of list pepe. Up to 7.25.4 they carried no rt_id: a filtered subscriber never got them, and an unfiltered one got every append once per live list of the topic. Since 7.25.5 the event is EVF_AUTHZ_SUBSCRIBE, and read carries the __subscribe_event__ alias. So when the yuno sets enable_subscription_authz, a remote subscription to the feed needs read on the tranger’s service, the permission that open-rt asks. Up to 7.25.4 a user without read could not open a feed, but could subscribe to the event with no filter and get every record of the feeds that other users opened. See YUNO_AUTH.md §4.6.

# the feed, and the subscription of the same session to its own records
command-yuno id=1911 service=tranger command=open-rt rt_id=rt1 topic_name=pp key=1

gobj_subscribe_event(gobj_remote_tranger, "EV_TRANGER_RECORD_ADDED", {
    __filter__: {rt_id: "rt1"}
}, gobj);

A handle is its opener’s: the channel it was opened by, and the user. The feeds, iterators and live lists that a remote session opens are that session’s. close-rt, close-iterator, close-list, get-page and get-list-data from ANOTHER session answer -403 “<role^name>: iterator ‘it1’ is not yours: another session opened it”, and log a warning, “Handle of another session, refused” (with the src and the user); the handle is left as it was. A gobj of the yuno itself (a local caller) is not refused. In 7.25.4 any session could close or read any handle by its id, and print-tranger shows the ids to every read user.

What “session” means depends on how the command arrives:

So through the agent another USER is refused, and a relayed command of no user is refused a handle that a user opened; two sessions of the SAME user through the agent are one owner, and each can close what the other opened -- the channel cannot tell them apart. The handles of a relayed command are not reaped when the operator’s session closes (only a direct session is watched, see open-iterator): close them.

# user alice, through the agent
ycommand -c 'command-yuno id=1911 service=tranger command=open-iterator iterator_id=it1 topic_name=pp key=1'
# user bob, through the agent, the same id
ycommand -c 'command-yuno id=1911 service=tranger command=close-iterator iterator_id=it1'
# -> -403: <role^name>: iterator 'it1' is not yours: another session opened it

The same holds, per connection, for two clients connected to the yuno directly: a GUI that opened it1 keeps it from another GUI, even one of the same user.

open-iterator match conditions (all optional, and 0 or empty means unset): from_t/to_t, from_tm/to_tm, from_rowid/to_rowid, backward, and the user_flag conditions (user_flag, not_user_flag, user_flag_mask_set, user_flag_mask_notset). They are ANDed, and every one is honored per record.

Several keys in one iterator. Give rkey instead of key. The iterator lays the matching keys end to end, sorted by key, in the same order tr2list prints a topic. get-page positions are positions in that concatenation. Each record gives its key in __md_tranger__.key, because one page can hold more than one key. The order is not a time order: a rowid counts inside one key only. To see the records by time, sort the page on the client. The match conditions apply to each key separately. key and rkey together is an error.

command-yuno id=<id> service=<tranger> command=open-iterator iterator_id=all1 topic_name=binaries rkey=.*
command-yuno id=<id> service=<tranger> command=get-page iterator_id=all1 from_rowid=1 limit=100
command-yuno id=<id> service=<tranger> command=close-iterator iterator_id=all1

The answer of open-iterator also gives keys, the number of keys it found.

A key deleted under it is a deleted key, born again or not. A multi-key iterator holds no tranger2 iterator between pages, so it keeps one realtime feed over the topic (only_md, fed to nobody) for its key_deleted callback: the next get-page after a delete-key of one of its keys answers -1 “was deleted” and closes the iterator, like a one-key iterator does, even when the key was created again in between. Open it again to page the reborn key. Only in the process that deleted the key (the master): the callback fires in the delete itself, so a REPLICA’s feed never hears it. There the next get-page reads files that are gone: it fails on the read, logs “Key gone from disk while its iterator was open: deleted (by the master?)”, and the iterator stays open. Close it and open it again.

An id is free again the moment its handle is gone. A handle goes with its topic (a close and reopen, delete-topic + create-topic, a stop/start of the service): the entry is dropped at the stop, after the delete-topic, when a get-page answers “closed with its topic”, or when open-iterator / open-rt / open-list meet the id again — so the same id opens a new handle, with data, instead of “already open” and no data.

The keys watch is not a client’s feed. It is named <iterator_id>^__keys__ and opened under a creator of its own, so a client’s open-rt rt_id=all1^__keys__ is another feed: until after 7.25.3 the two collided, open-iterator answered -1, and a get-page of a dead iterator closed the client’s feed. It records only the deletes of the keys its iterator pages.

What it costs. The open counts the rows of each key once (for a filtered iterator that is building and dropping each key’s index) and keeps only a number per key; a get-page opens the keys its page touches, reads, and closes them. Until after 7.24.1 it kept one tranger2 iterator per key for its whole life — 1000 keys with 60 daily files was +147 MB for one card. A key the iterator does not filter is counted LIVE at every get-page, like a one-key iterator: a backward page (newest first) shows the rows appended since the open. A filtered key keeps the count of the rows its filter matched at open (after 7.25.3; before, every key kept the count of the open, so the whole-topic card never showed a new row): a row appended to it after the open is never paged, although each page indexes the key again -- counting it would mean indexing every filtered key on every page. A key CREATED after the open is not in the iterator: open it again to see it.

Positions move under appends. Because an unfiltered key is counted live, a row appended to a key moves the positions after it by one. Paging forward after an append to an EARLIER key (or backward after an append to a LATER key), the next page repeats the last row of the page before; an append never makes a page skip a row. Every record carries its key and rowid, so a client that must not show a row twice drops a repeated (key, rowid). For example, with key I holding 3 rows and key J 2:

command-yuno id=<id> service=<tranger> command=open-iterator iterator_id=ij topic_name=t rkey=^[IJ]$
command-yuno id=<id> service=<tranger> command=get-page iterator_id=ij from_rowid=1 limit=4
#   I#1 I#2 I#3 J#1
#   (a row is appended to I)
command-yuno id=<id> service=<tranger> command=get-page iterator_id=ij from_rowid=5 limit=4
#   J#1 J#2         <- J#1 again

Bound the reading of a large topic anyway: a negative from_rowid reads the last N records of EACH key, and is what gui_treedb’s whole-topic card starts with:

command-yuno id=<id> service=<tranger> command=open-iterator iterator_id=all1 topic_name=binaries rkey=.* from_rowid=-100

A tranger record carries two independent timestamps, and a browser of raw records needs both:

They diverge whenever data is backfilled or a device uploads a buffered batch late. Both are expressed in the topic’s unit — seconds, unless its system_flag sets sf_t_ms / sf_tm_ms (ask topics expanded=1).

C_TREEDB¶

Hierarchical tree database manager — manages TreeDB instances on top of timeranger with JSON schema support.

Key attributes¶

Commands¶

Events. EV_OPEN_TREEDB and EV_CLOSE_TREEDB are public events of C_TREEDB, and they are NOT implemented: each one answers -1 and logs an ERROR, “EV_OPEN_TREEDB is not implemented: nothing opened, use the command open-treedb” (and the same for EV_CLOSE_TREEDB), with src and treedb_name. In 7.25.4 the open did nothing and answered 0, and neither said anything. Open and close a treedb with the commands, which ask their permission:

command-yuno id=<id> service=treedbs command=close-treedb treedb_name=treedb_foo force=1

C_NODE¶

Node resource interface for TreeDB — full CRUD and graph operations on tree nodes with linking, snapshots, and import/export.

Key attributes¶

Commands¶

Every comment starts with the yuno that answers (<role^name>: ), and a failure says its own cause and points at the log: create-node answers <role^name>: Node created, 'item01' of topic 'items' or <role^name>: cannot create the node of topic 'items' (see the log); shoot-snap, snap-content, nodes, instances, parents, children, jtree, desc, links, hooks, topics and import-assets the same (new after 7.25.4). In 7.25.4 they answered gobj_log_last_message(), the process-global buffer of the last ERROR of anybody -- a stale or unrelated cause -- and most answers carried no yuno. The exception is the refusal of a permission, “No permission to ‘<perm>’ in service ‘<treedb>’”, which names its service. instances answers -1 when it cannot list (it answered 0). The test walks C_NODE’s command table (tests/c/c_node_authz), so a command added with a bare comment fails it. The log says its own cause too: an update of a node that does not exist, with create and a create the library refuses, logs the library’s error and then “Cannot update node: it does not exist and it cannot be created (see the previous log)” (new after 7.25.4; 7.25.4 logged the last message again as its own).

Permissions¶

Every command that reads or writes the treedb asks for a permission, inside the command, with or without the global enable_command_authz gate. It matters only in a yuno with an authz checker (C_AUTHZ).

Only help and authzs ask nothing. system-schema, trace and the set-link-events display answered anyone until after 7.24.1; the test walks C_NODE’s command table (tests/c/c_node_authz), so a command added without a permission fails it.

read also guards the treedb’s feed: the five EV_TREEDB_NODE_* events are EVF_AUTHZ_SUBSCRIBE, and read carries the __subscribe_event__ alias, so in a yuno that sets enable_subscription_authz a peer needs read on the treedb service to subscribe to them (new after 7.25.4; in 7.25.4 any authenticated user got the feed). The gate is off by default, and it goes in the yuno’s config:

{
    "yuno": {
        "enable_subscription_authz": true
    }
}

A peer without read is refused and the channel stays open: ERROR “No permission to subscribe event”, with service, event, permission: read and username. See YUNO_AUTH.md §4.6 and the test tests/c/c_subscription_authz, which subscribes to a real C_NODE.

update-node with options.create=1 also asks for create when the node does not exist yet. A refusal answers -403, and nothing is written.

options.create_only=1 makes a NEW node and refuses one that exists (“Node already exists”); options.create=1 alone is an upsert, and a taken id there is an update of the existing record. A table’s +New sends create_only (gobj-ui 7.23.194). Both ask for create when the node does not exist: create_only alone used to create under update.

command-yuno id=<id> service=<treedb> command=update-node topic_name=users record='{"id":"bob","username":"Bob"}' options='{"create_only":1}'

An update-node with autolink whose record names a link that cannot be made saves the record and answers -1: “<role^name>: node ‘x’ saved, but some of its links were NOT changed: a link it names cannot be made, and that column was left as it was (see the log)”. The fkey column keeps the links it had (see treedb_replace_links()); it used to answer “Node update!”. The answer is the update’s own, even when a subscriber of its events updates the same service before it answers (until 7.25.3 the nested update reset it to a success).

ycommand -c 'command-yuno id=<id> service=<treedb> command=delete-node topic_name=items record={"id":"item1"} options={"ignore_snaps":1}'
# -403 No permission to 'create' in service '<treedb>'   (a role with 'delete' only)

A replica writes nothing, whoever asks. The commands answer “READ-ONLY” before anything is read. From C, every write method refuses before it moves anything: gobj_update_node() returns NULL for every update but a volatil one (memory only) -- the door C_AUTHZ uses for a user created from an event, which no command guards; gobj_link_nodes(), gobj_unlink_nodes() and gobj_delete_node() return -1 and log “Cannot link nodes / unlink nodes / delete a node on a READ-ONLY replica” (new after 7.25.4; 7.25.4 moved the links in memory first and met the refused save last); gobj_create_node() is refused by treedb_create_node(); and shooting or activating a snap by the treedb (“Only master can ...”).

/*  On a replica: -1, nothing moved  */
int ret = gobj_delete_node(gobj_node, "items",
    json_pack("{s:s}", "id", "item00"), json_pack("{s:b}", "force", 1), src);

An autolink write that fails at its save changes nothing it wrote. update-node (or gobj_update_node()) with autolink writes the fields, the links and the save as ONE write (treedb_update_node_and_links()). When the append fails (a full disk, a store gone), it answers NULL / -1. The fields and the links go back in memory to what the disk has, and none of the events of the write is told (EV_TREEDB_NODE_LINKED, EV_TREEDB_NODE_UPDATED):

• create + autolink: the create is on disk, with its ordinary fkeys EMPTY (a create stores none: only its file columns, which it links itself, and, for a secondary instance, the links it inherits from the primary). Its links are taken back, so memory and disk both have the node without them. The create was told (EV_TREEDB_NODE_CREATED), and an ERROR says what is left: “Node created, but its links cannot be saved (autolink): the node stays without them”.

• update + autolink: the node, its fields and its links are as the disk has them, in memory and on disk. One exception: the bytes of a file column of the record are stored BEFORE the write opens, as a write of their own. The blob and its __assets__ node stay on disk, and their EV_TREEDB_NODE_CREATED (or EV_TREEDB_NODE_UPDATED, for a new name of bytes already stored) is told. The node does not link them.

In 7.25.4 this write was three calls, and a failed save took nothing back: memory kept the fields and the links until the next reload, and EV_TREEDB_NODE_LINKED had been told. Write it again once the cause is fixed: the same update-node with autolink converges.

/*  the store of `users` cannot be written (a full disk)  */
json_t *node = gobj_update_node(gobj_node, "users",
    json_pack("{s:s, s:s, s:[s]}",
        "id", "alice",
        "username", "ALICE-NEW",
        "departments", "departments^direction^users"
    ),
    json_pack("{s:b}", "autolink", 1),
    src
);
/*  node == NULL; alice keeps her old username and links, in memory as on
 *  disk; direction does not hook her; no event was published            */

EV_TREEDB_UPDATE_NODE is an input event for gobjs of the SAME yuno (gobj_send_event()), with no permission asked. It is not a public event since after 7.25.3: C_IEVENT_CLI delivered a public event from its peer to the local service the event names, so the other end of an outbound session could write the treedb.

Example: a user whose only role on treedb_devices has "permission": "read" gets nodes and node, and is refused link-nodes:

ycommand -c 'command-yuno id=<yuno> service=treedb_devices command=link-nodes parent_ref=places^p1^devices child_ref=devices^d1'
# -403 No permission to 'update' in service 'treedb_devices'

File columns: __assets__¶

A treedb node often owns something that is not JSON — a photo, a plan, a signed pdf. Those bytes cannot go in the treedb: it is held in memory and timeranger2 rewrites the whole record on every update, so a 40 KB photo would be rewritten every time its node changed state and would ride along in every page of nodes. Measured on one census: 12 134 blobs, 346 MB on disk, would be ~460 MB of RAM for the life of the yuno.

So you mark a column file, and treedb gives you a pseudo-filesystem: you hand it a file, you get it back; the index lives in memory and the content on disk. Design note: DESIGN-treedb-files.md.

'foto': {'header': 'Photo', 'type': 'string', 'flag': ['fkey', 'file']}

• The column is an fkey into __assets__, a system topic every treedb creates at open next to __snaps__ and __graphs__ (shown in system mode only). Its rows are the index entries: id (the sha256 of the content), content_type, size, t, original_name (the only writable one), uploaded_by. Its hooks are derived: for every column C of topic T flagged file, __assets__ gains as_<T>_<C> -> {T: C} in memory, never written to topic_cols.json. The host declares nothing but the column, and nothing about __assets__ is ever versioned. The derivation follows the schema at run time: create-topic and delete-topic are live commands, so a topic added while the yuno runs gets its hook, and one deleted takes its hook — and the children it held — away with it.

• The bytes live under the treedb, at <treedb dir>/.blobs/ab/cd/<sha256>.<ext>, so a cp -a of the treedb directory carries the nodes and their bytes.

• The bytes ride beside the record, never inside the column, in a __files__ manifest keyed by column, consumed at the door. A browser sends content64; a C caller (command or EV_TREEDB_UPDATE_NODE alike) puts a real gbuffer in the kw and the manifest says which slice is whose (offset, size) — a kw holds ONE binary field, so one buffer carries every file of the record:

{"topic_name": "devices",
 "record": {"id": "E22000041",
            "foto": "",
            "__files__": {"foto": {"content64": "...", "original_name": "E22000041.jpg",
                                   "content_type": "image/jpeg"}}},
 "options": {"create": 1, "autolink": 1}}

• Treedb re-hashes what arrives. A client may put the sha256 in the column up front, ask node whether that asset exists and skip the bytes if it does (a census reload then sends ~0 instead of 346 MB) — but the id is a claim, never an authority: a wrong id with good bytes is refused, and a bare id of an asset nobody stored is refused. A bare id of an existing asset links it. A column may also arrive holding the full reference, and then it must be exactly __assets__^<id>^as_<T>_<C> of that very column: the link is made by the hook the value names, so one naming another file column’s hook would store the file into that other column.

• Size and type are checked at the door, on the bytes. The size is checked on the base64 before decoding; the type is sniffed from the first bytes and the declared one must agree — a png called image/jpeg is refused, and an svg called image/png is refused, which is the case the allowlist exists for. Two levels: the treedb’s ceiling (files_max_size, files_content_types) and the column’s policy, in its properties ({'max_size': 4096, 'content_types': ['application/pdf']}), which narrows the ceiling and never raises it. The ceiling itself sits behind the transport’s: the message was accepted whole and parsed before treedb saw it, so keep files_max_size under the transport’s max_pkt_size.

• Three writes, in order: the blob, the __assets__ node, the host record with its link. Interrupted early it leaves an orphan blob or an orphan index node, which gc-assets takes; never a link to nothing. A create-node whose file column cannot be linked is undone, so the answer never says yes over a record with an empty column.

• The write path links the file column itself, autolink or not. An ordinary fkey moves only through link-nodes or an autolink; a file column is edited by handing over a file, and the link is part of it: create-node and update-node link what the column names, "" unlinks, and a column the record does not carry is left alone. The autolink in the example above is for the OTHER fkeys of the record, not for foto.

• A second arrival of the same bytes is an update of the asset node, so the history of __assets__ says every name a file arrived under (a manifest that carries no original_name, or the name already stored, says nothing new about the file and appends nothing). The blob is written once, and the first arrival names it for ever: the extension is part of the served path and the URL is cached for ever, so a later arrival that declares another member of the same container (audio/mp4 for what was stored as video/mp4) keeps the stored content_type and says so in the log.

• gc-assets reads the snapshots, exactly. shoot-snap skips every __ topic, so an asset node never carries a tag and the tag guard never protects it: the collector walks the instances on disk of every topic with a file column and keeps what an ACTIVATION would load — per key, the newest instance under each existing snap’s tag, and only that one (a save is untagged since 7.24.0, so that instance is the record the snap SHOT, and a node that moves on after the shot does not drag the snap’s hold along). A snapshot holds bytes alive; deleting the snap frees them, and the delete is delete-node on its __snaps__ row (there is no delete-snap command). A treedb with no snapshot does not walk. It also takes the bytes with no row — what an interrupted write leaves, and the .tmp of one that never reached its rename — because every other reader of the store goes through the rows. .blobs is the tranger’s, so what counts as named is the union of every treedb’s __assets__.

• Deleting an asset node deletes its bytes. Refused while a node links it, like any parent with children — and refused, force included, while a snapshot links it: delete-node on an __assets__ row runs the same walk gc-assets does, because the ordinary tag guard never fires for an asset. force means “unlink the children”, never “ignore what a snapshot needs”, and no key of options skips the walk: the gc skips it through a private entry, not through anything the wire can spell. On a tranger that hosts more than one treedb, __assets__ and its bytes are shared: both gc-assets and delete-node read every treedb’s links, and an asset another treedb of the tranger links is refused, force or not.

• A file column must be ['fkey','file'] on a string, and that is checked in three places: at open (fatal), by create-topic with the yuno running (the topic is refused, the answer says why), and by the write path (the write is refused). A column that is file and not fkey would store its bytes into a column nothing links, and gc-assets would take them.

• ⚠️ A non-master replica gets the index and not the bytes. The watcher replicates a topic’s files, so a replica has every row of __assets__ and none of the .blobs behind them: get-asset answers “asset has no bytes on disk” there unless a web server in front is serving a copy of the blob directory. Getting the bytes to a replica is not solved — plan for it, or serve the assets from the master. The same holds for export-db / import-db: they carry the rows of __assets__ and not the .blobs.

Serving the bytes to a browser is C_ASSETS’ job.

C_RESOURCE2¶

Simple resource persistence — stores each resource as a flat JSON file.

Key attributes¶

C_ASSETS¶

The way out of the bytes a treedb keeps for its file columns: a signed URL a web server checks by itself, or the bytes inline when there is no web server in front. Storing is treedb’s (File columns): file columns, __assets__, and the import-assets / gc-assets commands of C_NODE. This gclass only publishes what is stored, and is one command.

The asset id is the sha256 of its content, so a served URL means the same bytes for ever and can be cached for ever. get-asset takes the asset id and nothing else — not a node plus a column: keyed by node, the same URL would mean different bytes the moment the node was relinked.

Key attributes¶

Commands¶

Gated by the read authz of the service.

get-asset takes asset_id, not id. command-yuno hands its whole kw to gobj_list_nodes() as the filter that picks the yuno, so a parameter named like a field of the yuno record becomes a filter on that field: an id of a sha256 matches no yuno and the answer is “Yuno not found” — which names the yuno and never the parameter. The bare id still works for a caller that never crosses the agent.

Two ways out to a browser, and the service picks¶

get-asset answers in one of two shapes:

{"mode": "url",    "url": "/media/ab/cd/<id>.jpg?e=<expires>&s=<token>"}
{"mode": "inline", "content_type": "image/jpeg", "content64": "..."}

It signs a URL when public_url and sign_secret are both configured, and answers inline when they are not. So the caller has one code path, and a node with no web server in front of it still shows its images instead of showing nothing.

The signed form reproduces, byte for byte, what this nginx block hashes. Do not use /assets/: a Vite SPA on the same vhost already owns that prefix for its content-hashed bundles, and the two locations would fight over it. The alias is the treedb’s own blob directory.

location /media/ {
    secure_link      $arg_s,$arg_e;
    secure_link_md5  "$secure_link_expires$uri <sign_secret>";
    if ($secure_link = "")  { return 403; }   # bad signature
    if ($secure_link = "0") { return 410; }   # expired
    alias <store>/<realm>/treedb_<name>/.blobs/;
    expires max;    # the name IS the hash: it can never go stale
    access_log off;
}

The client address is deliberately not in the signature: it would tie the URL to one IP and break every phone that changes network mid-session. The short lifetime is what limits a leaked URL.

secure_link needs nginx built --with-http_secure_link_module. Yuneta’s nginx and openresty are, but a node still running an older build must serve inline until its web server is replaced.

tests/c/c_assets covers the whole round trip: a record with its bytes beside it through update-node, both doors (content64 and a gbuffer of two slices), get-asset inline and signed, import-assets with hostile paths (.., absolute, a symlink out of the root), and gc-assets.