The Forum for Discussion about The Third Manifesto and Related Matters

Please or Register to create posts and topics.

The future of TTM

I know, so the benefit has to be in what Streams does not provide: enhanced queries, database, relvars, transactions, constraints etc. All the database stuff.

But you won't find interesting new solutions by looking at how Streams and LINQ are used now. You have to look for problems they couldn't solve, and where the SQL-based solution works but is clunky.

I'm not sure a (hopefully slightly-less-clunky) TTM-based approach to queries, databases, transactions and constraints is enough to overcome the momentum (or maybe it's inertia) of SQLite (or Berkeley DB, or other NoSQL dbs) and LINQ/Streams.

But maybe it is.

As I've been saying, shared database (RDBMS) and and non-database (LINQ/Streams) are off-topic. The on-topic space is:

  • in-process database
  • persistent, updatable
  • relational data model (tables, not objects or documents).

The list of competitors is:

  • SQLite (if SQL not an explicit requirement)
  • NoSQL (some, if RM)
  • Various ad hoc and roll-your-own (some, if RM).

What surprises me is how few choices there are. I recently did some searching for REST API server software, and I found well over 20 credible products spread across multiple languages. New languages, GUI frameworks, games engine, NoSQL, etc: dozens if not hundreds. There is no obvious reason why there are so many competitors in other spaces and so few in this one.

 

Andl - A New Database Language - andl.org
Quote from dandl on July 30, 2021, 5:11 am

I know, so the benefit has to be in what Streams does not provide: enhanced queries, database, relvars, transactions, constraints etc. All the database stuff.

But you won't find interesting new solutions by looking at how Streams and LINQ are used now. You have to look for problems they couldn't solve, and where the SQL-based solution works but is clunky.

I'm not sure a (hopefully slightly-less-clunky) TTM-based approach to queries, databases, transactions and constraints is enough to overcome the momentum (or maybe it's inertia) of SQLite (or Berkeley DB, or other NoSQL dbs) and LINQ/Streams.

But maybe it is.

As I've been saying, shared database (RDBMS) and and non-database (LINQ/Streams) are off-topic. The on-topic space is:

  • in-process database
  • persistent, updatable
  • relational data model (tables, not objects or documents).

The list of competitors is:

  • SQLite (if SQL not an explicit requirement)
  • NoSQL (some, if RM)
  • Various ad hoc and roll-your-own (some, if RM).

What surprises me is how few choices there are. I recently did some searching for REST API server software, and I found well over 20 credible products spread across multiple languages. New languages, GUI frameworks, games engine, NoSQL, etc: dozens if not hundreds. There is no obvious reason why there are so many competitors in other spaces and so few in this one.

I suppose it's no more or less surprising than the small number of true relational model implementations in the shared database world, and probably for the same reasons.

But if you look at it from a requirements point of view rather than a solution point of view -- i.e., in-process, persistent, updatable, query-able (for a very broad definition of "query") storage -- rather than specifically a relational model, then there are myriad choices.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org
Quote from Dave Voorhis on July 30, 2021, 9:30 am
Quote from dandl on July 30, 2021, 5:11 am

I know, so the benefit has to be in what Streams does not provide: enhanced queries, database, relvars, transactions, constraints etc. All the database stuff.

But you won't find interesting new solutions by looking at how Streams and LINQ are used now. You have to look for problems they couldn't solve, and where the SQL-based solution works but is clunky.

I'm not sure a (hopefully slightly-less-clunky) TTM-based approach to queries, databases, transactions and constraints is enough to overcome the momentum (or maybe it's inertia) of SQLite (or Berkeley DB, or other NoSQL dbs) and LINQ/Streams.

But maybe it is.

As I've been saying, shared database (RDBMS) and and non-database (LINQ/Streams) are off-topic. The on-topic space is:

  • in-process database
  • persistent, updatable
  • relational data model (tables, not objects or documents).

The list of competitors is:

  • SQLite (if SQL not an explicit requirement)
  • NoSQL (some, if RM)
  • Various ad hoc and roll-your-own (some, if RM).

What surprises me is how few choices there are. I recently did some searching for REST API server software, and I found well over 20 credible products spread across multiple languages. New languages, GUI frameworks, games engine, NoSQL, etc: dozens if not hundreds. There is no obvious reason why there are so many competitors in other spaces and so few in this one.

I suppose it's no more or less surprising than the small number of true relational model implementations in the shared database world, and probably for the same reasons.

But if you look at it from a requirements point of view rather than a solution point of view -- i.e., in-process, persistent, updatable, query-able (for a very broad definition of "query") storage -- rather than specifically a relational model, then there are myriad choices.

That got me thinking. Writing a rock-solid storage  engine with ACID compliance, multi-threaded, high performance is really hard. Grafting a relational API on top is relatively simple. So maybe the right place to  look is storage engines that are widely used but do not have a relational model, and then add one. Even if it already has an SQL layer, adding a relational API might well be of value.

The best list I found is this one: https://en.wikipedia.org/wiki/Embedded_database. Some support SQL, many don't, or only at a low level. A portable relational API that works across multiple engines -- an interesting idea.

Andl - A New Database Language - andl.org
Quote from dandl on July 31, 2021, 1:25 pm
Quote from Dave Voorhis on July 30, 2021, 9:30 am
Quote from dandl on July 30, 2021, 5:11 am

I know, so the benefit has to be in what Streams does not provide: enhanced queries, database, relvars, transactions, constraints etc. All the database stuff.

But you won't find interesting new solutions by looking at how Streams and LINQ are used now. You have to look for problems they couldn't solve, and where the SQL-based solution works but is clunky.

I'm not sure a (hopefully slightly-less-clunky) TTM-based approach to queries, databases, transactions and constraints is enough to overcome the momentum (or maybe it's inertia) of SQLite (or Berkeley DB, or other NoSQL dbs) and LINQ/Streams.

But maybe it is.

As I've been saying, shared database (RDBMS) and and non-database (LINQ/Streams) are off-topic. The on-topic space is:

  • in-process database
  • persistent, updatable
  • relational data model (tables, not objects or documents).

The list of competitors is:

  • SQLite (if SQL not an explicit requirement)
  • NoSQL (some, if RM)
  • Various ad hoc and roll-your-own (some, if RM).

What surprises me is how few choices there are. I recently did some searching for REST API server software, and I found well over 20 credible products spread across multiple languages. New languages, GUI frameworks, games engine, NoSQL, etc: dozens if not hundreds. There is no obvious reason why there are so many competitors in other spaces and so few in this one.

I suppose it's no more or less surprising than the small number of true relational model implementations in the shared database world, and probably for the same reasons.

But if you look at it from a requirements point of view rather than a solution point of view -- i.e., in-process, persistent, updatable, query-able (for a very broad definition of "query") storage -- rather than specifically a relational model, then there are myriad choices.

That got me thinking. Writing a rock-solid storage  engine with ACID compliance, multi-threaded, high performance is really hard. Grafting a relational API on top is relatively simple. So maybe the right place to  look is storage engines that are widely used but do not have a relational model, and then add one. Even if it already has an SQL layer, adding a relational API might well be of value.

The best list I found is this one: https://en.wikipedia.org/wiki/Embedded_database. Some support SQL, many don't, or only at a low level. A portable relational API that works across multiple engines -- an interesting idea.

Cool. Sounds like it might be a worthy pursuit.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org
Quote from dandl on July 31, 2021, 1:25 pm

That got me thinking. Writing a rock-solid storage  engine with ACID compliance, multi-threaded, high performance is really hard. Grafting a relational API on top is relatively simple. So maybe the right place to  look is storage engines that are widely used but do not have a relational model, and then add one. Even if it already has an SQL layer, adding a relational API might well be of value.

The best list I found is this one: https://en.wikipedia.org/wiki/Embedded_database. Some support SQL, many don't, or only at a low level. A portable relational API that works across multiple engines -- an interesting idea.

I agree with this proposal, and that is what I have been meaning to do for awhile now.

But it doesn't even have to be embedded.  I was thinking in terms of targeting layered DBMSs in general such that their main value is the lower level parts and one can put bespoke data models on top.

See https://en.wikipedia.org/wiki/FoundationDB for an example of what I'm talking about.  I was interested in that one since it was first announced, initially by a startup, which Apple then bought, but its still open source and being maintained.

 

Quote from dandl on July 31, 2021, 1:25 pm

 

That got me thinking. Writing a rock-solid storage  engine with ACID compliance, multi-threaded, high performance is really hard. Grafting a relational API on top is relatively simple. So maybe the right place to  look is storage engines that are widely used but do not have a relational model, and then add one. Even if it already has an SQL layer, adding a relational API might well be of value.

Then perhaps against all odds, it might be useful for you to keep on thinking.  It might even lead you to understand how "Writing a rock-solid storage engine with ACID compliance" is an utter manifestation of conflation of levels.  "Storage engine" is 100% physical level, "ACID compliance" has connotations, maybe even mostly, that pertain to the logical level.

And your gratuitious claim that "Grafting a relational API on top is relatively simple." is undeniable evidence that you still have serious problems looking at things "from a requirements point of view rather than a solution point of view".

Quote from Dave Voorhis on July 31, 2021, 6:49 pm
Quote from dandl on July 31, 2021, 1:25 pm
Quote from Dave Voorhis on July 30, 2021, 9:30 am
Quote from dandl on July 30, 2021, 5:11 am

I know, so the benefit has to be in what Streams does not provide: enhanced queries, database, relvars, transactions, constraints etc. All the database stuff.

But you won't find interesting new solutions by looking at how Streams and LINQ are used now. You have to look for problems they couldn't solve, and where the SQL-based solution works but is clunky.

I'm not sure a (hopefully slightly-less-clunky) TTM-based approach to queries, databases, transactions and constraints is enough to overcome the momentum (or maybe it's inertia) of SQLite (or Berkeley DB, or other NoSQL dbs) and LINQ/Streams.

But maybe it is.

As I've been saying, shared database (RDBMS) and and non-database (LINQ/Streams) are off-topic. The on-topic space is:

  • in-process database
  • persistent, updatable
  • relational data model (tables, not objects or documents).

The list of competitors is:

  • SQLite (if SQL not an explicit requirement)
  • NoSQL (some, if RM)
  • Various ad hoc and roll-your-own (some, if RM).

What surprises me is how few choices there are. I recently did some searching for REST API server software, and I found well over 20 credible products spread across multiple languages. New languages, GUI frameworks, games engine, NoSQL, etc: dozens if not hundreds. There is no obvious reason why there are so many competitors in other spaces and so few in this one.

I suppose it's no more or less surprising than the small number of true relational model implementations in the shared database world, and probably for the same reasons.

But if you look at it from a requirements point of view rather than a solution point of view -- i.e., in-process, persistent, updatable, query-able (for a very broad definition of "query") storage -- rather than specifically a relational model, then there are myriad choices.

That got me thinking. Writing a rock-solid storage  engine with ACID compliance, multi-threaded, high performance is really hard. Grafting a relational API on top is relatively simple. So maybe the right place to  look is storage engines that are widely used but do not have a relational model, and then add one. Even if it already has an SQL layer, adding a relational API might well be of value.

The best list I found is this one: https://en.wikipedia.org/wiki/Embedded_database. Some support SQL, many don't, or only at a low level. A portable relational API that works across multiple engines -- an interesting idea.

Cool. Sounds like it might be a worthy pursuit.

I think so, but so far I don't have a clear use case. Each of those products has a community of users and a community of contributors, and it takes time to learn and understand where the pain points are and what problems need solving. At this point what I see is the first real possibility of a useful application for real users with real problems not currently solved by SQL or Streams/Linq, and that's something.

Andl - A New Database Language - andl.org
Quote from Darren Duncan on July 31, 2021, 7:41 pm
Quote from dandl on July 31, 2021, 1:25 pm

That got me thinking. Writing a rock-solid storage  engine with ACID compliance, multi-threaded, high performance is really hard. Grafting a relational API on top is relatively simple. So maybe the right place to  look is storage engines that are widely used but do not have a relational model, and then add one. Even if it already has an SQL layer, adding a relational API might well be of value.

The best list I found is this one: https://en.wikipedia.org/wiki/Embedded_database. Some support SQL, many don't, or only at a low level. A portable relational API that works across multiple engines -- an interesting idea.

I agree with this proposal, and that is what I have been meaning to do for awhile now.

But it doesn't even have to be embedded.  I was thinking in terms of targeting layered DBMSs in general such that their main value is the lower level parts and one can put bespoke data models on top.

See https://en.wikipedia.org/wiki/FoundationDB for an example of what I'm talking about.  I was interested in that one since it was first announced, initially by a startup, which Apple then bought, but its still open source and being maintained.

 

Embedded was never a requirement, it was just a way to find a useful list of candidates. I agree, FoundationDB is a candidate. There are probably many others.

The key criteria are:

  • it's a database, not just about queries
  • it's got a low-latency API (in-process or close to it)
  • it does not have a server hosting SQL (or any query language)
  • it has real users with real problems that TTM-like features can solve.
Andl - A New Database Language - andl.org
Quote from Erwin on July 31, 2021, 9:12 pm
Quote from dandl on July 31, 2021, 1:25 pm

 

That got me thinking. Writing a rock-solid storage  engine with ACID compliance, multi-threaded, high performance is really hard. Grafting a relational API on top is relatively simple. So maybe the right place to  look is storage engines that are widely used but do not have a relational model, and then add one. Even if it already has an SQL layer, adding a relational API might well be of value.

Then perhaps against all odds, it might be useful for you to keep on thinking.  It might even lead you to understand how "Writing a rock-solid storage engine with ACID compliance" is an utter manifestation of conflation of levels.  "Storage engine" is 100% physical level, "ACID compliance" has connotations, maybe even mostly, that pertain to the logical level.

I disagree. InnoDB is describe as "a transaction-safe (ACID compliant) storage engine for MySQL". XtraDB, RocksDB, TokuDB and many others describe themselves as storage engines with ACID/transactional features. Storage engines are 2+ layers above physical. I do facts, not connotations.

And your gratuitious claim that "Grafting a relational API on top is relatively simple." is undeniable evidence that you still have serious problems looking at things "from a requirements point of view rather than a solution point of view".

I diagree, and I object strongly. You have absolutely no idea of my abilities outside the narrow confines of this forum, and my comment about a particular technical possibility bears no relationship to the quite separate concern of solving users' problems. This language is quite inappropriate for this forum and I ask you to desist.

Andl - A New Database Language - andl.org
Quote from dandl on August 1, 2021, 4:32 am
Quote from Erwin on July 31, 2021, 9:12 pm
Quote from dandl on July 31, 2021, 1:25 pm

That got me thinking. Writing a rock-solid storage  engine with ACID compliance, multi-threaded, high performance is really hard. Grafting a relational API on top is relatively simple. So maybe the right place to  look is storage engines that are widely used but do not have a relational model, and then add one. Even if it already has an SQL layer, adding a relational API might well be of value.

Then perhaps against all odds, it might be useful for you to keep on thinking.  It might even lead you to understand how "Writing a rock-solid storage engine with ACID compliance" is an utter manifestation of conflation of levels.  "Storage engine" is 100% physical level, "ACID compliance" has connotations, maybe even mostly, that pertain to the logical level.

I disagree. InnoDB is describe as "a transaction-safe (ACID compliant) storage engine for MySQL". XtraDB, RocksDB, TokuDB and many others describe themselves as storage engines with ACID/transactional features. Storage engines are 2+ layers above physical. I do facts, not connotations.

I agree with David.

Fundamentally ACID compliance can be conceived at a very low level, like at the low level of a transactional file system.

At its simplest, the entire database is a single sequence of octets, and all database update operations are just modifying the sequence of octets comprising the database.

ACID just enumerates features of being transactional, that an update to the octet sequence happens entirely or not at all, that each user doesn't see the intermediate state of a change by other users, that when the system reports a change was saved it is indeed saved, etc.

Any data model relational or otherwise is then just an abstraction over top of that, saying how to interpret the octets, which is fully distinct from the transactional/ACID qualities.

For the most part the underlying storage engine can represent the database how it wants, whether a single sequence of octets or a set of key-value pairs or whatever, my point remains, you can layer some other kind of model such as relational on top of it.

Given that providing transactions/ACID can be a tricky problem regardless of data model, it makes complete sense to let one vendor focus on the transactional/ACID stuff and another value add with the model, and they complement  each other, each vendor focuses on what they're better at.