Clarification re local relvar and KEY

#41 · June 10, 2020, 1:35 pm

Quote from dandl on June 10, 2020, 1:35 pm

For example, you might specify attribute types via type parameters -- which allows them to be virtually anything -- but require that they implement IComparable (C#?) or Comparable (Java) interfaces so that you can compare one attribute value to another.

This is the standard modern approach to implementing generic libraries like those for encryption or linear algebra. A good library can then be as universal as a given host language will allow, which means the same library can be equally applicable to a new language, an existing language's extension, or any other type of application.

Of course I know all that stuff, but generics and type parameters have limitations too (and believe me, I've pushed on them!). Generics are not enough on their own to solve the problem of headingsand interesting abstractions go further than that.

Yes, as you've discovered (and has often been discussed on this forum in the past), the TTM approach to headings -- and some operators themselves -- is incompatible with at least C# and Java's (and probably C++'s) static typing, which are all essentially the same model. But other approaches don't require giving up static typing guarantees, and that's why we have LINQ, Streams, etc. If you're going to code natively in C# or Java, they are preferable, because they don't require giving up static typing guarantees.

As I've been demonstrating, the gap is not so wide. You can't do a natural join in LINQ, or union on anything but the same type, or lots of other stuff. Andl.NET took that as far as it can go, but it doesn't go far enough.

Arguably, they don't give up anything else either -- at least in the context of C# or Java programming -- but they are certainly different from the TTM approach, as they are suited to their host languages.

The TTM approach is best suited to a very different kind of language.

No, not that different. Value types are within reach, that just leaves headings. The rest is easy.

It's not the basic RA let alone the ERA, and it's been done to death. I'm interested in breaking new ground.

As a side note, once you get rid of open expressions, it leaves a query language that can be built visually. No coding, just string together relops, headings, functions, data sources and sinks. But that's a topic for another time.

A new query language that involves no coding is, of course, a completely different thing.

Here, we're talking about implementing the relational model in a mainstream programming language. In that context, even if we set aside some of the other abominations that Erwin mentioned (like pointers/references, nulls, etc., in a D), giving up static typing is a fundamental obstacle. If you're going to treat heading manipulation as some sort of dynamic operation, then it's in the same category as another embedded mini-language -- regular expressions -- with all their attendant blecherousness, along with the grandaddy of embedded dynamic language horrors we'd all like to avoid: SQL.

I don't think so. If you set your sights low, subset the RA and rely on native records and hand-coded RA-like operations you get LINQ or Streams. It's useful, but it's been done. I did that with Andl.Net and learned a lot about C# generics. I also found it's really limiting.

For me, the goal is higher: the extended RA, not a subset; in fact I've been calling it AndlERA. That is something I don't think has been done.

When you target a higher level abstraction you run into limitations on the languages, but you open up new territory. You can solve the problems of data interchange between Java-only programs using only the features of Java, but you better enjoy the island because you're stuck on it. You can talk about how nicely you've abstracted your Java classes, but it's still Java. Thrift supports upwards of 20 languages. SQL works not because it's SQL but because of ODBC, callable in just about every known language.

But I'm still exploring the territory, so it's hard to say where it will lead.

For example, you might specify attribute types via type parameters -- which allows them to be virtually anything -- but require that they implement IComparable (C#?) or Comparable (Java) interfaces so that you can compare one attribute value to another.

This is the standard modern approach to implementing generic libraries like those for encryption or linear algebra. A good library can then be as universal as a given host language will allow, which means the same library can be equally applicable to a new language, an existing language's extension, or any other type of application.

Of course I know all that stuff, but generics and type parameters have limitations too (and believe me, I've pushed on them!). Generics are not enough on their own to solve the problem of headingsand interesting abstractions go further than that.

Yes, as you've discovered (and has often been discussed on this forum in the past), the TTM approach to headings -- and some operators themselves -- is incompatible with at least C# and Java's (and probably C++'s) static typing, which are all essentially the same model. But other approaches don't require giving up static typing guarantees, and that's why we have LINQ, Streams, etc. If you're going to code natively in C# or Java, they are preferable, because they don't require giving up static typing guarantees.

As I've been demonstrating, the gap is not so wide. You can't do a natural join in LINQ, or union on anything but the same type, or lots of other stuff. Andl.NET took that as far as it can go, but it doesn't go far enough.

Arguably, they don't give up anything else either -- at least in the context of C# or Java programming -- but they are certainly different from the TTM approach, as they are suited to their host languages.

The TTM approach is best suited to a very different kind of language.

No, not that different. Value types are within reach, that just leaves headings. The rest is easy.

It's not the basic RA let alone the ERA, and it's been done to death. I'm interested in breaking new ground.

As a side note, once you get rid of open expressions, it leaves a query language that can be built visually. No coding, just string together relops, headings, functions, data sources and sinks. But that's a topic for another time.

A new query language that involves no coding is, of course, a completely different thing.

Here, we're talking about implementing the relational model in a mainstream programming language. In that context, even if we set aside some of the other abominations that Erwin mentioned (like pointers/references, nulls, etc., in a D), giving up static typing is a fundamental obstacle. If you're going to treat heading manipulation as some sort of dynamic operation, then it's in the same category as another embedded mini-language -- regular expressions -- with all their attendant blecherousness, along with the grandaddy of embedded dynamic language horrors we'd all like to avoid: SQL.

I don't think so. If you set your sights low, subset the RA and rely on native records and hand-coded RA-like operations you get LINQ or Streams. It's useful, but it's been done. I did that with Andl.Net and learned a lot about C# generics. I also found it's really limiting.

For me, the goal is higher: the extended RA, not a subset; in fact I've been calling it AndlERA. That is something I don't think has been done.

When you target a higher level abstraction you run into limitations on the languages, but you open up new territory. You can solve the problems of data interchange between Java-only programs using only the features of Java, but you better enjoy the island because you're stuck on it. You can talk about how nicely you've abstracted your Java classes, but it's still Java. Thrift supports upwards of 20 languages. SQL works not because it's SQL but because of ODBC, callable in just about every known language.

But I'm still exploring the territory, so it's hard to say where it will lead.

Andl - A New Database Language - andl.org

#42 · June 10, 2020, 4:43 pm

Quote from Dave Voorhis on June 10, 2020, 4:43 pm

Quote from dandl on June 10, 2020, 1:35 pm

For example, you might specify attribute types via type parameters -- which allows them to be virtually anything -- but require that they implement IComparable (C#?) or Comparable (Java) interfaces so that you can compare one attribute value to another.

This is the standard modern approach to implementing generic libraries like those for encryption or linear algebra. A good library can then be as universal as a given host language will allow, which means the same library can be equally applicable to a new language, an existing language's extension, or any other type of application.

Of course I know all that stuff, but generics and type parameters have limitations too (and believe me, I've pushed on them!). Generics are not enough on their own to solve the problem of headingsand interesting abstractions go further than that.

Yes, as you've discovered (and has often been discussed on this forum in the past), the TTM approach to headings -- and some operators themselves -- is incompatible with at least C# and Java's (and probably C++'s) static typing, which are all essentially the same model. But other approaches don't require giving up static typing guarantees, and that's why we have LINQ, Streams, etc. If you're going to code natively in C# or Java, they are preferable, because they don't require giving up static typing guarantees.

As I've been demonstrating, the gap is not so wide. You can't do a natural join in LINQ, or union on anything but the same type, or lots of other stuff. Andl.NET took that as far as it can go, but it doesn't go far enough.

We've discussed natural joins before. Of course, these seem like limitations if you're coming at querying from a TTM point of view; not much of a limitation if you're approaching querying from a more traditionally functional-programming inspired point of view. Yes, the lack of JOIN is an issue -- the assumption in C# and Java being that you're querying on the already-existing object graph of a running program, rather than needing to create an object graph on ad-hoc data -- but it's straightforward to add JOIN mechanisms or use existing ones. In the Java world, jOOλ is probably the best-known tool to address this. See https://www.baeldung.com/jool section 5.2

I presume there are equivalents for C#.

Arguably, they don't give up anything else either -- at least in the context of C# or Java programming -- but they are certainly different from the TTM approach, as they are suited to their host languages.

The TTM approach is best suited to a very different kind of language.

No, not that different. Value types are within reach, that just leaves headings. The rest is easy.

So far, "easy" seems to require dispensing with static type guarantees. I suppose some might consider that a fair tradeoff. I don't.

It's not the basic RA let alone the ERA, and it's been done to death. I'm interested in breaking new ground.

As a side note, once you get rid of open expressions, it leaves a query language that can be built visually. No coding, just string together relops, headings, functions, data sources and sinks. But that's a topic for another time.

A new query language that involves no coding is, of course, a completely different thing.

Here, we're talking about implementing the relational model in a mainstream programming language. In that context, even if we set aside some of the other abominations that Erwin mentioned (like pointers/references, nulls, etc., in a D), giving up static typing is a fundamental obstacle. If you're going to treat heading manipulation as some sort of dynamic operation, then it's in the same category as another embedded mini-language -- regular expressions -- with all their attendant blecherousness, along with the grandaddy of embedded dynamic language horrors we'd all like to avoid: SQL.

I don't think so. If you set your sights low, subset the RA and rely on native records and hand-coded RA-like operations you get LINQ or Streams. It's useful, but it's been done. I did that with Andl.Net and learned a lot about C# generics. I also found it's really limiting.

For me, the goal is higher: the extended RA, not a subset; in fact I've been calling it AndlERA. That is something I don't think has been done.

When you target a higher level abstraction you run into limitations on the languages, but you open up new territory. You can solve the problems of data interchange between Java-only programs using only the features of Java, but you better enjoy the island because you're stuck on it. You can talk about how nicely you've abstracted your Java classes, but it's still Java. Thrift supports upwards of 20 languages. SQL works not because it's SQL but because of ODBC, callable in just about every known language.

But I'm still exploring the territory, so it's hard to say where it will lead.

I'm not sure what Thrift has to do with it -- you mean the RPC library? -- but I'm not sure why you think Java is limited to Java, but other languages aren't limited to... Other languages...? ??

Rel happily maps CSV files, ODBC/JDBC SQL DBMSs, MS Access, MS Excel and its native storage from its own database and remote Rel databases to relvars -- but also provides Java base classes and interfaces to permit adding other data sources of any kind in a straightforward fashion. That is the benefit of coding to appropriate levels of abstraction in Java (or any programming language.)

So it's certainly not limited to Java in some fashion. It's written in Java, but that doesn't limit data to the Java ecosystem; it only means it leverages the Java ecosystem to develop core functionality.

Quote from dandl on June 10, 2020, 1:35 pm

For example, you might specify attribute types via type parameters -- which allows them to be virtually anything -- but require that they implement IComparable (C#?) or Comparable (Java) interfaces so that you can compare one attribute value to another.

This is the standard modern approach to implementing generic libraries like those for encryption or linear algebra. A good library can then be as universal as a given host language will allow, which means the same library can be equally applicable to a new language, an existing language's extension, or any other type of application.

Of course I know all that stuff, but generics and type parameters have limitations too (and believe me, I've pushed on them!). Generics are not enough on their own to solve the problem of headingsand interesting abstractions go further than that.

Yes, as you've discovered (and has often been discussed on this forum in the past), the TTM approach to headings -- and some operators themselves -- is incompatible with at least C# and Java's (and probably C++'s) static typing, which are all essentially the same model. But other approaches don't require giving up static typing guarantees, and that's why we have LINQ, Streams, etc. If you're going to code natively in C# or Java, they are preferable, because they don't require giving up static typing guarantees.

As I've been demonstrating, the gap is not so wide. You can't do a natural join in LINQ, or union on anything but the same type, or lots of other stuff. Andl.NET took that as far as it can go, but it doesn't go far enough.

We've discussed natural joins before. Of course, these seem like limitations if you're coming at querying from a TTM point of view; not much of a limitation if you're approaching querying from a more traditionally functional-programming inspired point of view. Yes, the lack of JOIN is an issue -- the assumption in C# and Java being that you're querying on the already-existing object graph of a running program, rather than needing to create an object graph on ad-hoc data -- but it's straightforward to add JOIN mechanisms or use existing ones. In the Java world, jOOλ is probably the best-known tool to address this. See https://www.baeldung.com/jool section 5.2

I presume there are equivalents for C#.

Arguably, they don't give up anything else either -- at least in the context of C# or Java programming -- but they are certainly different from the TTM approach, as they are suited to their host languages.

The TTM approach is best suited to a very different kind of language.

No, not that different. Value types are within reach, that just leaves headings. The rest is easy.

So far, "easy" seems to require dispensing with static type guarantees. I suppose some might consider that a fair tradeoff. I don't.

It's not the basic RA let alone the ERA, and it's been done to death. I'm interested in breaking new ground.

As a side note, once you get rid of open expressions, it leaves a query language that can be built visually. No coding, just string together relops, headings, functions, data sources and sinks. But that's a topic for another time.

A new query language that involves no coding is, of course, a completely different thing.

Here, we're talking about implementing the relational model in a mainstream programming language. In that context, even if we set aside some of the other abominations that Erwin mentioned (like pointers/references, nulls, etc., in a D), giving up static typing is a fundamental obstacle. If you're going to treat heading manipulation as some sort of dynamic operation, then it's in the same category as another embedded mini-language -- regular expressions -- with all their attendant blecherousness, along with the grandaddy of embedded dynamic language horrors we'd all like to avoid: SQL.

I don't think so. If you set your sights low, subset the RA and rely on native records and hand-coded RA-like operations you get LINQ or Streams. It's useful, but it's been done. I did that with Andl.Net and learned a lot about C# generics. I also found it's really limiting.

For me, the goal is higher: the extended RA, not a subset; in fact I've been calling it AndlERA. That is something I don't think has been done.

When you target a higher level abstraction you run into limitations on the languages, but you open up new territory. You can solve the problems of data interchange between Java-only programs using only the features of Java, but you better enjoy the island because you're stuck on it. You can talk about how nicely you've abstracted your Java classes, but it's still Java. Thrift supports upwards of 20 languages. SQL works not because it's SQL but because of ODBC, callable in just about every known language.

But I'm still exploring the territory, so it's hard to say where it will lead.

I'm not sure what Thrift has to do with it -- you mean the RPC library? -- but I'm not sure why you think Java is limited to Java, but other languages aren't limited to... Other languages...? ??

Rel happily maps CSV files, ODBC/JDBC SQL DBMSs, MS Access, MS Excel and its native storage from its own database and remote Rel databases to relvars -- but also provides Java base classes and interfaces to permit adding other data sources of any kind in a straightforward fashion. That is the benefit of coding to appropriate levels of abstraction in Java (or any programming language.)

So it's certainly not limited to Java in some fashion. It's written in Java, but that doesn't limit data to the Java ecosystem; it only means it leverages the Java ecosystem to develop core functionality.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org

#43 · June 11, 2020, 1:24 am

Quote from dandl on June 11, 2020, 1:24 am
We've discussed natural joins before. Of course, these seem like limitations if you're coming at querying from a TTM point of view; not much of a limitation if you're approaching querying from a more traditionally functional-programming inspired point of view. Yes, the lack of JOIN is an issue -- the assumption in C# and Java being that you're querying on the already-existing object graph of a running program, rather than needing to create an object graph on ad-hoc data -- but it's straightforward to add JOIN mechanisms or use existing ones. In the Java world, jOOλ is probably the best-known tool to address this. See https://www.baeldung.com/jool section 5.2

I presume there are equivalents for C#.

Natural joins are just an exemplar. The point is, you can't translate SQL queries, or TD queries such as those in DTATRM, or RA examples you might find on the web, directly into any widely used language and get type safety. Given the ERA, type safety and existing GP language, you get to pick any two.

Arguably, they don't give up anything else either -- at least in the context of C# or Java programming -- but they are certainly different from the TTM approach, as they are suited to their host languages.

The TTM approach is best suited to a very different kind of language.

No, not that different. Value types are within reach, that just leaves headings. The rest is easy.

So far, "easy" seems to require dispensing with static type guarantees. I suppose some might consider that a fair tradeoff. I don't.

For me, the goal is higher: the extended RA, not a subset; in fact I've been calling it AndlERA. That is something I don't think has been done.

When you target a higher level abstraction you run into limitations on the languages, but you open up new territory. You can solve the problems of data interchange between Java-only programs using only the features of Java, but you better enjoy the island because you're stuck on it. You can talk about how nicely you've abstracted your Java classes, but it's still Java. Thrift supports upwards of 20 languages. SQL works not because it's SQL but because of ODBC, callable in just about every known language.

But I'm still exploring the territory, so it's hard to say where it will lead.

I'm not sure what Thrift has to do with it -- you mean the RPC library? -- but I'm not sure why you think Java is limited to Java, but other languages aren't limited to... Other languages...? ??

What I'm trying to get across is the idea that there are abstractions at a higher level than those found in any single programming language. You cling to type safety in Java, but what do you do about types Java doesn't have? Java is only 'universal' on the JVM island; ditto C# and the CLR; ditto Python and so on.

There are just a few examples of 'universal' coding abstractions that are like bridges between islands. The ones that come to mind are JSON, ODBC and Thrift, but there are a few others. ODBC as an abstraction deals with tables and rows and columns, but as an implementation deals with strings and integers. In C# it's type safe at the API level, but the types corresponding to tables are not exposed through the interface.

Rel happily maps CSV files, ODBC/JDBC SQL DBMSs, MS Access, MS Excel and its native storage from its own database and remote Rel databases to relvars -- but also provides Java base classes and interfaces to permit adding other data sources of any kind in a straightforward fashion. That is the benefit of coding to appropriate levels of abstraction in Java (or any programming language.)

So it's certainly not limited to Java in some fashion. It's written in Java, but that doesn't limit data to the Java ecosystem; it only means it leverages the Java ecosystem to develop core functionality.

Of course you do, and good luck to you. Andl does exactly the same and we should be happy on our little islands. But we have no way of exposing our type systems to programs outside the JVM/CLR islands, and no way of enforcing at compile time the type safety of stuff we import, such as CSV, ODBC, SQL etc.

A successful AndlERA project would allow my C# (or Python) program to to perform RA queries on your Rel (or Java) database without knowing or writing any code in any of your languages.

We've discussed natural joins before. Of course, these seem like limitations if you're coming at querying from a TTM point of view; not much of a limitation if you're approaching querying from a more traditionally functional-programming inspired point of view. Yes, the lack of JOIN is an issue -- the assumption in C# and Java being that you're querying on the already-existing object graph of a running program, rather than needing to create an object graph on ad-hoc data -- but it's straightforward to add JOIN mechanisms or use existing ones. In the Java world, jOOλ is probably the best-known tool to address this. See https://www.baeldung.com/jool section 5.2

I presume there are equivalents for C#.

Natural joins are just an exemplar. The point is, you can't translate SQL queries, or TD queries such as those in DTATRM, or RA examples you might find on the web, directly into any widely used language and get type safety. Given the ERA, type safety and existing GP language, you get to pick any two.

Arguably, they don't give up anything else either -- at least in the context of C# or Java programming -- but they are certainly different from the TTM approach, as they are suited to their host languages.

The TTM approach is best suited to a very different kind of language.

No, not that different. Value types are within reach, that just leaves headings. The rest is easy.

So far, "easy" seems to require dispensing with static type guarantees. I suppose some might consider that a fair tradeoff. I don't.

For me, the goal is higher: the extended RA, not a subset; in fact I've been calling it AndlERA. That is something I don't think has been done.

When you target a higher level abstraction you run into limitations on the languages, but you open up new territory. You can solve the problems of data interchange between Java-only programs using only the features of Java, but you better enjoy the island because you're stuck on it. You can talk about how nicely you've abstracted your Java classes, but it's still Java. Thrift supports upwards of 20 languages. SQL works not because it's SQL but because of ODBC, callable in just about every known language.

But I'm still exploring the territory, so it's hard to say where it will lead.

I'm not sure what Thrift has to do with it -- you mean the RPC library? -- but I'm not sure why you think Java is limited to Java, but other languages aren't limited to... Other languages...? ??

What I'm trying to get across is the idea that there are abstractions at a higher level than those found in any single programming language. You cling to type safety in Java, but what do you do about types Java doesn't have? Java is only 'universal' on the JVM island; ditto C# and the CLR; ditto Python and so on.

There are just a few examples of 'universal' coding abstractions that are like bridges between islands. The ones that come to mind are JSON, ODBC and Thrift, but there are a few others. ODBC as an abstraction deals with tables and rows and columns, but as an implementation deals with strings and integers. In C# it's type safe at the API level, but the types corresponding to tables are not exposed through the interface.

Rel happily maps CSV files, ODBC/JDBC SQL DBMSs, MS Access, MS Excel and its native storage from its own database and remote Rel databases to relvars -- but also provides Java base classes and interfaces to permit adding other data sources of any kind in a straightforward fashion. That is the benefit of coding to appropriate levels of abstraction in Java (or any programming language.)

So it's certainly not limited to Java in some fashion. It's written in Java, but that doesn't limit data to the Java ecosystem; it only means it leverages the Java ecosystem to develop core functionality.

Of course you do, and good luck to you. Andl does exactly the same and we should be happy on our little islands. But we have no way of exposing our type systems to programs outside the JVM/CLR islands, and no way of enforcing at compile time the type safety of stuff we import, such as CSV, ODBC, SQL etc.

A successful AndlERA project would allow my C# (or Python) program to to perform RA queries on your Rel (or Java) database without knowing or writing any code in any of your languages.

Andl - A New Database Language - andl.org

The Forum for Discussion about The Third Manifesto and Related Matters

Clarification re local relvar and KEY