Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

SData

Structured Data (SData) is a mechanism to define and manage structured fields, attributes, and commands in a hierarchical and schema-driven manner. It is used to define the attributes of objects, command parameters, and database-like records in a highly structured and reusable way.


Core Concepts

SData Fields

SData fields are descriptors that define individual fields or attributes. These fields include information about their type, name, default value, description, flags, and other properties.

SData Tables

SData tables are arrays of field descriptors (sdata_desc_t) that define structured data. These tables allow hierarchical definitions. This enables the creation of complex schemas.


Data Types (data_type_t)

The data_type_t enumeration defines the types of data that SData fields can represent:

TypeDescription
DTP_STRINGA text string.
DTP_BOOLEANA boolean value (TRUE or FALSE).
DTP_INTEGERAn integer value.
DTP_REALA floating-point number.
DTP_LISTA list (array) of values.
DTP_DICTA dictionary (key-value pairs).
DTP_JSONA JSON object.
DTP_POINTERA generic pointer.

Data Type Utilities


Field Flags (sdata_flag_t)

The sdata_flag_t enumeration defines the properties and characteristics of each field. Flags are bitwise-combinable to give fields multiple properties.

FlagDescription
SDF_NOTACCESSField is not accessible.
SDF_RDField is read-only.
SDF_WRField is writable (and readable).
SDF_REQUIREDField is required. It must not be null.
SDF_PERSISTField is persistent and must be saved/loaded.
SDF_VOLATILField is volatile and must not be saved/loaded.
SDF_RESOURCEField is a resource, referencing another schema.
SDF_PKEYField is a primary key.
SDF_STATSField holds statistical data (metadata).
SDF_RSTATSField holds resettable statistics, implicitly SDF_STATS.
SDF_PSTATSField holds persistent statistics, implicitly SDF_STATS.
SDF_AUTHZ_RRead access requires authorization (__read_attribute__).
SDF_AUTHZ_WWrite access requires authorization (__write_attribute__).
SDF_AUTHZ_XExecution requires authorization (__execute_command__).
SDF_AUTHZ_PAuthorization constraint parameter.
SDF_AUTHZ_SStats read requires authorization (__read_stats__).
SDF_AUTHZ_RSStats reset requires authorization (__reset_stats__).
SDF_SECRETA secret: read and saved as usual, SHOWN as ********.

SDF_NOTACCESS

Field is not accessible.

SDF_RD

Field is read-only.

SDF_WR

Field is writable (and readable). Only an attribute with SDF_WR can be
written at run time by `write-attr` (`gobj_is_writable_attr()`).

SDF_REQUIRED

Field is required. It must not be null.

SDF_PERSIST

Field is persistent and must be saved/loaded. It does not make the
field writable by `write-attr`: without SDF_WR it is set by the config,
or by the gclass's own command, which checks the value (up to 7.25.22
SDF_PERSIST alone was writable, so `write-attr` went round the checks
of such a command).
SDATA (DTP_INTEGER, "max_sessions_per_user", SDF_PERSIST,        "0", "config or set-max-sessions"),
SDATA (DTP_INTEGER, "cert_sync_interval_sec",SDF_WR|SDF_PERSIST, "900", "config or write-attr"),

SDF_VOLATIL

Field is volatile and must not be saved/loaded.

SDF_RESOURCE

Field is a resource, referencing another schema.

SDF_PKEY

Field is a primary key.

SDF_STATS

Field holds statistical data (metadata).

SDF_RSTATS

Field holds resettable statistics, implicitly `SDF_STATS`.

SDF_PSTATS

Field holds persistent statistics, implicitly `SDF_STATS`.

SDF_AUTHZ_R

Read access requires authorization (`__read_attribute__`).

SDF_AUTHZ_W

Write access requires authorization (`__write_attribute__`).

SDF_AUTHZ_X

Execution requires authorization (`__execute_command__`).

SDF_AUTHZ_S

Stats read requires authorization (`__read_stats__`).

SDF_AUTHZ_P

Authorization constraint parameter.

SDF_AUTHZ_RS

Stats reset requires authorization (`__reset_stats__`).

SDF_SECRET

A secret -- a password, a client secret, a token. Since 7.25.19. The
attr is read, written and persisted exactly as without the flag (the
emailsender still sends its real password, the persistent-attrs file
still keeps it); what changes is what is SHOWN: `view-attrs`,
`write-attr`, `list-persistent-attrs`, `view-gobj`, `view-config`, the
start-up trace of the yuno's attrs and the `create_delete2` trace of a
gobj being built answer `********` for it, whatever its json type (a
number or a boolean too). An absent value or an empty string is shown
as it is, so "not set" still reads as such. A dict of attrs is masked with
[`gobj_mask_secret_attrs()`](#gobj_mask_secret_attrs), a whole
configuration with [`gobj_mask_secret_config()`](#gobj_mask_secret_config).
The C_TCP `traffic` dump prints the bytes on the wire: a frame that
carries a credential is sent with `"__secret__": true` in the kw of
`EV_TX_DATA` (or marked with [`gbuffer_set_secret()`](#gbuffer_set_secret))
and is dumped as `<N bytes hidden>`.
SDATA (DTP_STRING, "password", SDF_PERSIST|SDF_SECRET, "", "email password"),
> command-yuno id=1 service=__yuno__ command=view-attrs attribute=password
{ "C_EMAILSENDER^emailsender": "********" }
The persistent-attrs file itself (`<realm>/<yuno>/data/*-persistent-attrs.json`)
is written 0600 since 7.25.19: up to 7.25.18 it took the process umask,
0666 on every node, and it holds these secrets in clear. A file of
the yuno's own that is not as a save writes it -- left wider by an
older release, or a hard link -- is REPLACED when it is loaded, by a
0600 one with the same content (logged as *"Persistent attrs file
replaced by a 0600 one of the yuno's own"*), not only at its next
save; nothing is changed through its other names. A file of another
user is left as it is, with a warning: a load only reads.
A symlink in place of the file is not read (*"Refused the persistent
attrs file: it is a symlink"*).

A save writes a NEW file in the same directory (`<file>.tmp-XXXXXX`,
created 0600 with `O_EXCL`), syncs it, and renames it over the old
one: a hard link or a symlink in its place is replaced (nothing is
written through it), and the old file is never truncated before the
new one is complete -- a save that fails leaves it as it was. A
`<file>.tmp-XXXXXX` left by a save that did not end is removed at the
next load, logged (one that is not a regular file is left, logged).

Whose file it is after a save: the yuno's user's. A file there of
another user was loaded first, so its attrs are in the save and
nothing is lost: run as the yuno's user, the save takes it over
(logged at INFO with the old owner); run as root (once, to debug), the
new file is given to the old owner, so root never takes the file from
the yuno. A file there that CANNOT be read refuses the save (its other
attrs would be lost): remove it (its attrs go back to their defaults)
or repair it. An empty file is no data and refuses nothing.

In a directory the yuno cannot write the save goes IN PLACE, into a
file of the yuno's own, regular and of one name only: room is reserved
first without growing the file, a shorter content is padded with blanks
and cut only after it is on disk, and a write that stops half way (a
full disk where the room could not be reserved, or on a copy-on-write
filesystem) writes the old content back. A crash from the write until
its sync returns, or a write back that fails too, can leave a file that
cannot be parsed -- the "never truncated" above holds for the rename,
not for this path -- and that file refuses the next saves as above. `write-attr` answers a save that fails
(*"<gobj>: <attr> written, but NOT saved (see the log)"*, `result`
-1); up to 7.25.20 it answered "done" with nothing on disk. A
persistent attr of a gobj that is no service is written and NOT
persisted, and the answer says so (*"..., NOT persisted (only a
service persists its attrs)"*, with a warning in the log).

A COMMAND PARAMETER takes the flag too: the `commands` trace prints
the command line, and with `ev_kw` its kw, with the parameter masked
(see [`command_mask_secret_kw()`](#command_mask_secret_kw)). A key the
command table cannot know -- the free keys of a `SDF_WILD_CMD` command,
which the agent's `command-yuno` forwards to another yuno -- is masked
when its NAME is a secret's ([`is_secret_name()`](#is_secret_name)).
SDATAPM (DTP_STRING,    "password",     SDF_SECRET,     0,          "Password"),
🌀🌀 mach(C_AUTHZ^authz), cmd: set-user-pwd username=bob password=********
🌀🌀 mach(C_AGENT^agent), cmd: command-yuno id=1 service=authz command=set-user-pwd password=********

Common Flag Combinations


Descriptor Fields (sdata_desc_t)

The sdata_desc_t structure defines a field or schema. Each descriptor specifies the following:

FieldTypeDescription
typedata_type_tThe type of the field (for example string, boolean).
nameconst char *The name of the field.
aliasconst char **Alternative names (aliases) for the field.
flagsdata_flag_tFlags defining the field’s properties.
default_valueconst char *The default value of the field.
headerconst char *Header text for table columns.
fillspaceintColumn width for table formatting.
descriptionconst char *A description of the field’s purpose.
json_fnjson_function_fnCustom function for processing JSON data.
schemaconst sdata_desc_t *Pointer to a sub-schema for compound fields.
authpthconst char *Authorization path for accessing or modifying the field.

Application

Attributes

SData tables define attributes by listing fields with their types, default values, and flags. These fields form the basis of object definitions. This enables schema-based validation and management.

Commands

SData tables can define commands with associated parameters, schemas, and descriptions. Commands extend the function of objects, providing structured inputs and outputs.

Nested Schemas

Fields in SData can reference other schemas. This enables hierarchical definitions. This allows for the creation of complex, nested structures while maintaining clarity and reusability.