How to resolve the type system-relational algebra mismatch
Quote from dandl on October 29, 2019, 1:07 pmQuote from Dave Voorhis on October 29, 2019, 8:37 amQuote from dandl on October 29, 2019, 12:21 amWe talk about the Object Relational Mismatch, but TTM shows us the problem is deeper than that. TTM/D presents a perfectly plausible type system in two parts:
- A scalar type system (RM Pre 4) reasonably similar to and compatible with value types (but not OO types) in other languages
- A non-scalar type system (RM Pre 6 and 7) designed specifically for the convenience of the 'usual operators of the relational algebra' (RM Pre 18), and highly incompatible with just about every other language.
The solutions on offer are:
- Implement TTM/D, as an island language with no bridges to anywhere. Write applications in industrial D. [This is the thrust of D&D]
- Implement TTM/D as a hosted language. Write applications in a mix of HL and D. [This is Andl, or Rel]
- Ignore TTM, implement a simple type system and a hackneyed version of the RA in a unique language (SQL). Write applications in a mix of HL and SQL, with some kind of bridge between them. [This is ORM, xDBC, +/- stored procedures]
- (variant on 3): embed or generate SQL in HL [This is LINQ to SQL, jOOQ, etc].
- Implement a modestly powerful type system and a simplified version of the RA in a minimal language, embed in HL. [This was the proposal I described recently, based on domains and JSON.]
- Find a new compromise: the intersection of a rich type system and a 'logical equivalent of the relation algebra' that is compatible with existing languages and can manage data held locally or on remote servers. [A dream?]
- Find a new direction: some method other than a novel type system to embed a 'logical equivalent of the relation algebra' into existing languages.
It seems (1) is a dead-end. I've done (2) and it's not going anywhere that I can see. The world has gone for 3 and 4, but not without pain. I know how to do (5), and I think it has possibilities.
But (6) and (7) would perhaps be more interesting. Thoughts?
6 and 7 are indeed more interesting, but lead to an inevitable question: Why do we want RA?
5 sounds like 3 but with a potentially awkward abstraction layer around SQL. It doesn't sound appealing, at least not to me.
No, it's not intended that way. It could at one extreme be viewed as a compiled version of Hibernate, with full compile-time type safety. Or as an ORM with less of the O, since it is designed to bring relations into the application, not push object persistence out onto an unwilling RDBMS. But I have work to do to see if it really has legs. [I found after a bit of a review, rather less choice of multi-language ORMs than I had found, and I know writing direct to ODBC is not a lot of fun.]
7 is purely speculative, so we can only imagine what novel and as-yet-unconsidered formalisms and practical approaches might be appropriate here.
I suggested one: represent relation/tuple types by simple classes but use inheritance/polymorphism to avoid type checking them, then run a post-build step to figure out the attributes. Post-process instead of pre-process. Andl for .NET kind of does that, and it works quite well for the most part.
That leaves 6.
Does 6 need -- or so strongly benefit from -- RA, such that there is virtually no alternative?
I would argue that if we're talking writing host-language code (for the masses, usually C# or Java or Python) we are almost certainly better off writing queries to data providers in native SQL + prepared statements via API calls, or MongoDB via API calls, or Hadoop API calls, or Apache POI/XXE (MS Excel) API calls, or whatever API calls.
Attempts to abstract these under one unifying API usually winds up with disappointing lack of capability ("the connector for Hadoop won't let me do x and I need x!", etc.) and/or grossly suboptimal code and/or non-support for the latest data provider until somebody writes a "connector" for the new data provider so that it's supported by the unifying API. Of course, the "connector" for that new data provider usually turns out to be inadequate to support some key feature of the new data provider, so the "connector" API needs to be changed so that every data provider supports it... And so on.
I think you're rationalising a gut feel. Those who are desperate to tune their programs to use the latest and greatest features of platform X will always have to turn to the SDK for X, and give up Y and Z. They were never my target.
I would hope to find a niche amongst people who want to write applications of modest scale across multiple platforms without ever learning very much about any of them. Breadth, not depth. Find a sample that goes close and hack it until it's close enough.
We're almost certainly better off not trying to unify all data providers under one "data" API. It may be tempting to turn data providers and their data into relvars, but I think that will end in a #4-like frustration.
So no RA there.
Data providers, when queried via their API's, typically emit collections -- usually (if Java) some Set<T> or List<T> or Iterator<T>, or similar. I suggest -- and I genuinely suggest, this is not a rhetorical "suggest" that is actually an attempt to persuade -- that for manipulation of result-of-query collections, we don't need the RA. We already have a suitable composable, expressive, container/collection abstraction: .NET LINQ or Java Streams. These provide all the abstraction and expressive composability that we seek from the RA, fully-integrated with the rich(ish) type system of the host language, and both are capable and familiar to Java/C# programmers. Yes, it's not the RA we're used to, and it requires hand-optimisation of queries, but I'm not sure that's a compelling reason to switch back to the RA and give up the clean integration that LINQ/Streams give us.
I agree, to a point. What LINQ does it does well, I would be lost without it. The DSL form was designed to build on the same idea: give devs a familiar SQL-like tool; as it happens I only ever use the API version.
But LINQ to SQL is not such a short step.Create the database, run a tool, generate a DBML, rinse, repeat (it's been a while). Updates use a side channel. And I don't think there is one for Java, just jOOQ? I haven't checked recently.
So the niche there is (a) at least as good as LINQ to SQL (b) does schemas and updates (c) simpler easier lighter (d) portable. There must be a few punters in there somewhere!
It does suggest there's maybe work to be done on automated re-writing of LINQ/Streams queries to provide query optimisation.
So where does that leave the RA?
Unfortunately, given these options, it leaves it behind. It's really 1 and 2 that we want -- and that's where the RA would be most effective and most applicable -- but that's not what happened.
The RA has headspace -- a lot of people have been taught it, or can easily learn it. And a feature allowing experts to tune the generated SQL would give it another thing LINQ doesn't have.
I'm not totally sold on the idea, but it does seem it might have some legs.
Quote from Dave Voorhis on October 29, 2019, 8:37 amQuote from dandl on October 29, 2019, 12:21 amWe talk about the Object Relational Mismatch, but TTM shows us the problem is deeper than that. TTM/D presents a perfectly plausible type system in two parts:
- A scalar type system (RM Pre 4) reasonably similar to and compatible with value types (but not OO types) in other languages
- A non-scalar type system (RM Pre 6 and 7) designed specifically for the convenience of the 'usual operators of the relational algebra' (RM Pre 18), and highly incompatible with just about every other language.
The solutions on offer are:
- Implement TTM/D, as an island language with no bridges to anywhere. Write applications in industrial D. [This is the thrust of D&D]
- Implement TTM/D as a hosted language. Write applications in a mix of HL and D. [This is Andl, or Rel]
- Ignore TTM, implement a simple type system and a hackneyed version of the RA in a unique language (SQL). Write applications in a mix of HL and SQL, with some kind of bridge between them. [This is ORM, xDBC, +/- stored procedures]
- (variant on 3): embed or generate SQL in HL [This is LINQ to SQL, jOOQ, etc].
- Implement a modestly powerful type system and a simplified version of the RA in a minimal language, embed in HL. [This was the proposal I described recently, based on domains and JSON.]
- Find a new compromise: the intersection of a rich type system and a 'logical equivalent of the relation algebra' that is compatible with existing languages and can manage data held locally or on remote servers. [A dream?]
- Find a new direction: some method other than a novel type system to embed a 'logical equivalent of the relation algebra' into existing languages.
It seems (1) is a dead-end. I've done (2) and it's not going anywhere that I can see. The world has gone for 3 and 4, but not without pain. I know how to do (5), and I think it has possibilities.
But (6) and (7) would perhaps be more interesting. Thoughts?
6 and 7 are indeed more interesting, but lead to an inevitable question: Why do we want RA?
5 sounds like 3 but with a potentially awkward abstraction layer around SQL. It doesn't sound appealing, at least not to me.
No, it's not intended that way. It could at one extreme be viewed as a compiled version of Hibernate, with full compile-time type safety. Or as an ORM with less of the O, since it is designed to bring relations into the application, not push object persistence out onto an unwilling RDBMS. But I have work to do to see if it really has legs. [I found after a bit of a review, rather less choice of multi-language ORMs than I had found, and I know writing direct to ODBC is not a lot of fun.]
7 is purely speculative, so we can only imagine what novel and as-yet-unconsidered formalisms and practical approaches might be appropriate here.
I suggested one: represent relation/tuple types by simple classes but use inheritance/polymorphism to avoid type checking them, then run a post-build step to figure out the attributes. Post-process instead of pre-process. Andl for .NET kind of does that, and it works quite well for the most part.
That leaves 6.
Does 6 need -- or so strongly benefit from -- RA, such that there is virtually no alternative?
I would argue that if we're talking writing host-language code (for the masses, usually C# or Java or Python) we are almost certainly better off writing queries to data providers in native SQL + prepared statements via API calls, or MongoDB via API calls, or Hadoop API calls, or Apache POI/XXE (MS Excel) API calls, or whatever API calls.
Attempts to abstract these under one unifying API usually winds up with disappointing lack of capability ("the connector for Hadoop won't let me do x and I need x!", etc.) and/or grossly suboptimal code and/or non-support for the latest data provider until somebody writes a "connector" for the new data provider so that it's supported by the unifying API. Of course, the "connector" for that new data provider usually turns out to be inadequate to support some key feature of the new data provider, so the "connector" API needs to be changed so that every data provider supports it... And so on.
I think you're rationalising a gut feel. Those who are desperate to tune their programs to use the latest and greatest features of platform X will always have to turn to the SDK for X, and give up Y and Z. They were never my target.
I would hope to find a niche amongst people who want to write applications of modest scale across multiple platforms without ever learning very much about any of them. Breadth, not depth. Find a sample that goes close and hack it until it's close enough.
We're almost certainly better off not trying to unify all data providers under one "data" API. It may be tempting to turn data providers and their data into relvars, but I think that will end in a #4-like frustration.
So no RA there.
Data providers, when queried via their API's, typically emit collections -- usually (if Java) some Set<T> or List<T> or Iterator<T>, or similar. I suggest -- and I genuinely suggest, this is not a rhetorical "suggest" that is actually an attempt to persuade -- that for manipulation of result-of-query collections, we don't need the RA. We already have a suitable composable, expressive, container/collection abstraction: .NET LINQ or Java Streams. These provide all the abstraction and expressive composability that we seek from the RA, fully-integrated with the rich(ish) type system of the host language, and both are capable and familiar to Java/C# programmers. Yes, it's not the RA we're used to, and it requires hand-optimisation of queries, but I'm not sure that's a compelling reason to switch back to the RA and give up the clean integration that LINQ/Streams give us.
I agree, to a point. What LINQ does it does well, I would be lost without it. The DSL form was designed to build on the same idea: give devs a familiar SQL-like tool; as it happens I only ever use the API version.
But LINQ to SQL is not such a short step.Create the database, run a tool, generate a DBML, rinse, repeat (it's been a while). Updates use a side channel. And I don't think there is one for Java, just jOOQ? I haven't checked recently.
So the niche there is (a) at least as good as LINQ to SQL (b) does schemas and updates (c) simpler easier lighter (d) portable. There must be a few punters in there somewhere!
It does suggest there's maybe work to be done on automated re-writing of LINQ/Streams queries to provide query optimisation.
So where does that leave the RA?
Unfortunately, given these options, it leaves it behind. It's really 1 and 2 that we want -- and that's where the RA would be most effective and most applicable -- but that's not what happened.
The RA has headspace -- a lot of people have been taught it, or can easily learn it. And a feature allowing experts to tune the generated SQL would give it another thing LINQ doesn't have.
I'm not totally sold on the idea, but it does seem it might have some legs.
Quote from Dave Voorhis on October 29, 2019, 2:06 pmQuote from dandl on October 29, 2019, 1:07 pmQuote from Dave Voorhis on October 29, 2019, 8:37 amQuote from dandl on October 29, 2019, 12:21 amWe talk about the Object Relational Mismatch, but TTM shows us the problem is deeper than that. TTM/D presents a perfectly plausible type system in two parts:
- A scalar type system (RM Pre 4) reasonably similar to and compatible with value types (but not OO types) in other languages
- A non-scalar type system (RM Pre 6 and 7) designed specifically for the convenience of the 'usual operators of the relational algebra' (RM Pre 18), and highly incompatible with just about every other language.
The solutions on offer are:
- Implement TTM/D, as an island language with no bridges to anywhere. Write applications in industrial D. [This is the thrust of D&D]
- Implement TTM/D as a hosted language. Write applications in a mix of HL and D. [This is Andl, or Rel]
- Ignore TTM, implement a simple type system and a hackneyed version of the RA in a unique language (SQL). Write applications in a mix of HL and SQL, with some kind of bridge between them. [This is ORM, xDBC, +/- stored procedures]
- (variant on 3): embed or generate SQL in HL [This is LINQ to SQL, jOOQ, etc].
- Implement a modestly powerful type system and a simplified version of the RA in a minimal language, embed in HL. [This was the proposal I described recently, based on domains and JSON.]
- Find a new compromise: the intersection of a rich type system and a 'logical equivalent of the relation algebra' that is compatible with existing languages and can manage data held locally or on remote servers. [A dream?]
- Find a new direction: some method other than a novel type system to embed a 'logical equivalent of the relation algebra' into existing languages.
It seems (1) is a dead-end. I've done (2) and it's not going anywhere that I can see. The world has gone for 3 and 4, but not without pain. I know how to do (5), and I think it has possibilities.
But (6) and (7) would perhaps be more interesting. Thoughts?
6 and 7 are indeed more interesting, but lead to an inevitable question: Why do we want RA?
5 sounds like 3 but with a potentially awkward abstraction layer around SQL. It doesn't sound appealing, at least not to me.
No, it's not intended that way. It could at one extreme be viewed as a compiled version of Hibernate, with full compile-time type safety. Or as an ORM with less of the O, since it is designed to bring relations into the application, not push object persistence out onto an unwilling RDBMS. But I have work to do to see if it really has legs. [I found after a bit of a review, rather less choice of multi-language ORMs than I had found, and I know writing direct to ODBC is not a lot of fun.]
I'm not a fan of Hibernate. I now use a thin skin wrapper around JDBC (ODBC for Java, basically) that uses a simple templating mechanism to replace SQL table names and makes it easy to create prepared statements and get back result sets. It's lean and simple, and I like it. I'm going to push it -- along with some related gadgetry -- out as open source before long.
Quote from dandl on October 29, 2019, 1:07 pmQuote from Dave Voorhis on October 29, 2019, 8:37 am7 is purely speculative, so we can only imagine what novel and as-yet-unconsidered formalisms and practical approaches might be appropriate here.
I suggested one: represent relation/tuple types by simple classes but use inheritance/polymorphism to avoid type checking them, then run a post-build step to figure out the attributes. Post-process instead of pre-process. Andl for .NET kind of does that, and it works quite well for the most part.
Yes, post-compile-time, pre-runtime "static" checking is a reasonable approach. I was thinking of #7 having more of a purely speculative, things-we-haven't-thought-of-yet, relatively bluesky-thinking orientation than that ("RA?!? Old stuff -- let's base this on categories and morphisms!"), but it seems notionally reasonable.
Quote from dandl on October 29, 2019, 1:07 pmI think you're rationalising a gut feel.
No, bitter and repeated experience with heavy wrappers and the like.
Quote from dandl on October 29, 2019, 1:07 pmBut LINQ to SQL is not such a short step.Create the database, run a tool, generate a DBML, rinse, repeat (it's been a while). Updates use a side channel. And I don't think there is one for Java, just jOOQ? I haven't checked recently.
I vaguely recall other attempts, but I can't find them now. (Not that I've looked very thoroughly.) There may be cultural resistance in the Java community, which seems to be trending toward embracing SQL rather than wrapping it away to invisibility.
Quote from dandl on October 29, 2019, 1:07 pmThe RA has headspace -- a lot of people have been taught it, or can easily learn it. And a feature allowing experts to tune the generated SQL would give it another thing LINQ doesn't have.
But is there anything inherent in the RA that gives us expressive power -- or intuitiveness -- that LINQ or Streams doesn't have?
There's no explicit notion of projection per se but it isn't required in LINQ/Streams (there are alternatives); selection is equivalent; EXTEND is map (well, not really, but winds up being used that way); aggregation is fold or collect and various Collectors (in Java; I forget the equivalent in LINQ); set operators are there. JOIN has equivalents in LINQ but seems a bit chewy (but do-able) in Streams -- I'll have to revisit that -- and so on. In short, LINQ and Streams seem equivalent to RA in expressive power, just different.
Being able to tune generated SQL sounds good, but presumably isn't bound to an RA implementation.
Quote from dandl on October 29, 2019, 1:07 pmQuote from Dave Voorhis on October 29, 2019, 8:37 amQuote from dandl on October 29, 2019, 12:21 amWe talk about the Object Relational Mismatch, but TTM shows us the problem is deeper than that. TTM/D presents a perfectly plausible type system in two parts:
- A scalar type system (RM Pre 4) reasonably similar to and compatible with value types (but not OO types) in other languages
- A non-scalar type system (RM Pre 6 and 7) designed specifically for the convenience of the 'usual operators of the relational algebra' (RM Pre 18), and highly incompatible with just about every other language.
The solutions on offer are:
- Implement TTM/D, as an island language with no bridges to anywhere. Write applications in industrial D. [This is the thrust of D&D]
- Implement TTM/D as a hosted language. Write applications in a mix of HL and D. [This is Andl, or Rel]
- Ignore TTM, implement a simple type system and a hackneyed version of the RA in a unique language (SQL). Write applications in a mix of HL and SQL, with some kind of bridge between them. [This is ORM, xDBC, +/- stored procedures]
- (variant on 3): embed or generate SQL in HL [This is LINQ to SQL, jOOQ, etc].
- Implement a modestly powerful type system and a simplified version of the RA in a minimal language, embed in HL. [This was the proposal I described recently, based on domains and JSON.]
- Find a new compromise: the intersection of a rich type system and a 'logical equivalent of the relation algebra' that is compatible with existing languages and can manage data held locally or on remote servers. [A dream?]
- Find a new direction: some method other than a novel type system to embed a 'logical equivalent of the relation algebra' into existing languages.
It seems (1) is a dead-end. I've done (2) and it's not going anywhere that I can see. The world has gone for 3 and 4, but not without pain. I know how to do (5), and I think it has possibilities.
But (6) and (7) would perhaps be more interesting. Thoughts?
6 and 7 are indeed more interesting, but lead to an inevitable question: Why do we want RA?
5 sounds like 3 but with a potentially awkward abstraction layer around SQL. It doesn't sound appealing, at least not to me.
No, it's not intended that way. It could at one extreme be viewed as a compiled version of Hibernate, with full compile-time type safety. Or as an ORM with less of the O, since it is designed to bring relations into the application, not push object persistence out onto an unwilling RDBMS. But I have work to do to see if it really has legs. [I found after a bit of a review, rather less choice of multi-language ORMs than I had found, and I know writing direct to ODBC is not a lot of fun.]
I'm not a fan of Hibernate. I now use a thin skin wrapper around JDBC (ODBC for Java, basically) that uses a simple templating mechanism to replace SQL table names and makes it easy to create prepared statements and get back result sets. It's lean and simple, and I like it. I'm going to push it -- along with some related gadgetry -- out as open source before long.
Quote from dandl on October 29, 2019, 1:07 pmQuote from Dave Voorhis on October 29, 2019, 8:37 am7 is purely speculative, so we can only imagine what novel and as-yet-unconsidered formalisms and practical approaches might be appropriate here.
I suggested one: represent relation/tuple types by simple classes but use inheritance/polymorphism to avoid type checking them, then run a post-build step to figure out the attributes. Post-process instead of pre-process. Andl for .NET kind of does that, and it works quite well for the most part.
Yes, post-compile-time, pre-runtime "static" checking is a reasonable approach. I was thinking of #7 having more of a purely speculative, things-we-haven't-thought-of-yet, relatively bluesky-thinking orientation than that ("RA?!? Old stuff -- let's base this on categories and morphisms!"), but it seems notionally reasonable.
Quote from dandl on October 29, 2019, 1:07 pmI think you're rationalising a gut feel.
No, bitter and repeated experience with heavy wrappers and the like.
Quote from dandl on October 29, 2019, 1:07 pmBut LINQ to SQL is not such a short step.Create the database, run a tool, generate a DBML, rinse, repeat (it's been a while). Updates use a side channel. And I don't think there is one for Java, just jOOQ? I haven't checked recently.
I vaguely recall other attempts, but I can't find them now. (Not that I've looked very thoroughly.) There may be cultural resistance in the Java community, which seems to be trending toward embracing SQL rather than wrapping it away to invisibility.
Quote from dandl on October 29, 2019, 1:07 pmThe RA has headspace -- a lot of people have been taught it, or can easily learn it. And a feature allowing experts to tune the generated SQL would give it another thing LINQ doesn't have.
But is there anything inherent in the RA that gives us expressive power -- or intuitiveness -- that LINQ or Streams doesn't have?
There's no explicit notion of projection per se but it isn't required in LINQ/Streams (there are alternatives); selection is equivalent; EXTEND is map (well, not really, but winds up being used that way); aggregation is fold or collect and various Collectors (in Java; I forget the equivalent in LINQ); set operators are there. JOIN has equivalents in LINQ but seems a bit chewy (but do-able) in Streams -- I'll have to revisit that -- and so on. In short, LINQ and Streams seem equivalent to RA in expressive power, just different.
Being able to tune generated SQL sounds good, but presumably isn't bound to an RA implementation.
Quote from dandl on October 29, 2019, 11:52 pmQuote from Dave Voorhis on October 29, 2019, 2:06 pmNo, it's not intended that way. It could at one extreme be viewed as a compiled version of Hibernate, with full compile-time type safety. Or as an ORM with less of the O, since it is designed to bring relations into the application, not push object persistence out onto an unwilling RDBMS. But I have work to do to see if it really has legs. [I found after a bit of a review, rather less choice of multi-language ORMs than I had found, and I know writing direct to ODBC is not a lot of fun.]
I'm not a fan of Hibernate. I now use a thin skin wrapper around JDBC (ODBC for Java, basically) that uses a simple templating mechanism to replace SQL table names and makes it easy to create prepared statements and get back result sets. It's lean and simple, and I like it. I'm going to push it -- along with some related gadgetry -- out as open source before long.
Again, I think Hibernate has a specific target audience, which is not everyone. The sticking points I've run into (in general) are:
- A lot of DBMS work managing the schema, adding initial data, versioning. Rails was pretty good at this part, the MS tools are not. This is avoided in a corporate environments (someone else does it) but front and centre in small standalone applications.
- No good path to getting strongly typed data transfer objects that match the SQL schema. Dynamic languages avoid the problem, in C#/Java you have to choose code first (generate SQL) or database first (generate code), and they both smell.
I've written about domains and the RA, but really it's a DSL mini-language based on familiar concepts, hopefully to address both.
Yes, post-compile-time, pre-runtime "static" checking is a reasonable approach. I was thinking of #7 having more of a purely speculative, things-we-haven't-thought-of-yet, relatively bluesky-thinking orientation than that ("RA?!? Old stuff -- let's base this on categories and morphisms!"), but it seems notionally reasonable.
Perhaps I should set my sights higher? ;-) But seriously, it was trying to riff on the idea of using the RA (or something very like it) without making a heading (as a set of attributes) part of the type system. How to take simple data objects in the code, relations/tables in the database, connect them together, and perform RA-like operations along the way?
But is there anything inherent in the RA that gives us expressive power -- or intuitiveness -- that LINQ or Streams doesn't have?
There's no explicit notion of projection per se but it isn't required in LINQ/Streams (there are alternatives); selection is equivalent; EXTEND is map (well, not really, but winds up being used that way); aggregation is fold or collect and various Collectors (in Java; I forget the equivalent in LINQ); set operators are there. JOIN has equivalents in LINQ but seems a bit chewy (but do-able) in Streams -- I'll have to revisit that -- and so on. In short, LINQ and Streams seem equivalent to RA in expressive power, just different.
They both describe a pipeline of roughly equivalent operations on collections of tuples. In a sense, LINQ is an RA. But LINQ-to-SQL suffers from the sticking points I listed above, and I think they're inherent. The idea is that a separate DSL provides an avenue to address those.
Being able to tune generated SQL sounds good, but presumably isn't bound to an RA implementation.
I could do my own LINQ-to-SQL implementation based on the MS docs, but where's the fun in that? The current idea is really just:
- stripped down Andl language with tweaks to the syntax
- simplified type system
- reuse the internal relational engine and SQL generation
- add some C#/Java code generation.
The hard parts are
- open expressions (including aggregation)
- what API to present to the developer.
Quote from Dave Voorhis on October 29, 2019, 2:06 pmNo, it's not intended that way. It could at one extreme be viewed as a compiled version of Hibernate, with full compile-time type safety. Or as an ORM with less of the O, since it is designed to bring relations into the application, not push object persistence out onto an unwilling RDBMS. But I have work to do to see if it really has legs. [I found after a bit of a review, rather less choice of multi-language ORMs than I had found, and I know writing direct to ODBC is not a lot of fun.]
I'm not a fan of Hibernate. I now use a thin skin wrapper around JDBC (ODBC for Java, basically) that uses a simple templating mechanism to replace SQL table names and makes it easy to create prepared statements and get back result sets. It's lean and simple, and I like it. I'm going to push it -- along with some related gadgetry -- out as open source before long.
Again, I think Hibernate has a specific target audience, which is not everyone. The sticking points I've run into (in general) are:
- A lot of DBMS work managing the schema, adding initial data, versioning. Rails was pretty good at this part, the MS tools are not. This is avoided in a corporate environments (someone else does it) but front and centre in small standalone applications.
- No good path to getting strongly typed data transfer objects that match the SQL schema. Dynamic languages avoid the problem, in C#/Java you have to choose code first (generate SQL) or database first (generate code), and they both smell.
I've written about domains and the RA, but really it's a DSL mini-language based on familiar concepts, hopefully to address both.
Yes, post-compile-time, pre-runtime "static" checking is a reasonable approach. I was thinking of #7 having more of a purely speculative, things-we-haven't-thought-of-yet, relatively bluesky-thinking orientation than that ("RA?!? Old stuff -- let's base this on categories and morphisms!"), but it seems notionally reasonable.
Perhaps I should set my sights higher? ;-) But seriously, it was trying to riff on the idea of using the RA (or something very like it) without making a heading (as a set of attributes) part of the type system. How to take simple data objects in the code, relations/tables in the database, connect them together, and perform RA-like operations along the way?
But is there anything inherent in the RA that gives us expressive power -- or intuitiveness -- that LINQ or Streams doesn't have?
There's no explicit notion of projection per se but it isn't required in LINQ/Streams (there are alternatives); selection is equivalent; EXTEND is map (well, not really, but winds up being used that way); aggregation is fold or collect and various Collectors (in Java; I forget the equivalent in LINQ); set operators are there. JOIN has equivalents in LINQ but seems a bit chewy (but do-able) in Streams -- I'll have to revisit that -- and so on. In short, LINQ and Streams seem equivalent to RA in expressive power, just different.
They both describe a pipeline of roughly equivalent operations on collections of tuples. In a sense, LINQ is an RA. But LINQ-to-SQL suffers from the sticking points I listed above, and I think they're inherent. The idea is that a separate DSL provides an avenue to address those.
Being able to tune generated SQL sounds good, but presumably isn't bound to an RA implementation.
I could do my own LINQ-to-SQL implementation based on the MS docs, but where's the fun in that? The current idea is really just:
- stripped down Andl language with tweaks to the syntax
- simplified type system
- reuse the internal relational engine and SQL generation
- add some C#/Java code generation.
The hard parts are
- open expressions (including aggregation)
- what API to present to the developer.
Quote from AntC on October 30, 2019, 3:03 amQuote from Dave Voorhis on October 29, 2019, 2:06 pmQuote from dandl on October 29, 2019, 1:07 pmQuote from Dave Voorhis on October 29, 2019, 8:37 amQuote from dandl on October 29, 2019, 12:21 amWe talk about the Object Relational Mismatch, but TTM shows us the problem is deeper than that. TTM/D presents a perfectly plausible type system in two parts:
- A scalar type system (RM Pre 4) reasonably similar to and compatible with value types (but not OO types) in other languages
- A non-scalar type system (RM Pre 6 and 7) designed specifically for the convenience of the 'usual operators of the relational algebra' (RM Pre 18), and highly incompatible with just about every other language.
The solutions on offer are:
- Implement TTM/D, as an island language with no bridges to anywhere. Write applications in industrial D. [This is the thrust of D&D]
- Implement TTM/D as a hosted language. Write applications in a mix of HL and D. [This is Andl, or Rel]
- Ignore TTM, implement a simple type system and a hackneyed version of the RA in a unique language (SQL). Write applications in a mix of HL and SQL, with some kind of bridge between them. [This is ORM, xDBC, +/- stored procedures]
- (variant on 3): embed or generate SQL in HL [This is LINQ to SQL, jOOQ, etc].
- Implement a modestly powerful type system and a simplified version of the RA in a minimal language, embed in HL. [This was the proposal I described recently, based on domains and JSON.]
- Find a new compromise: the intersection of a rich type system and a 'logical equivalent of the relation algebra' that is compatible with existing languages and can manage data held locally or on remote servers. [A dream?]
- Find a new direction: some method other than a novel type system to embed a 'logical equivalent of the relation algebra' into existing languages.
It seems (1) is a dead-end. I've done (2) and it's not going anywhere that I can see. The world has gone for 3 and 4, but not without pain. I know how to do (5), and I think it has possibilities.
But (6) and (7) would perhaps be more interesting. Thoughts?
6 and 7 are indeed more interesting, but lead to an inevitable question: Why do we want RA?
5 sounds like 3 but with a potentially awkward abstraction layer around SQL. It doesn't sound appealing, at least not to me.
Quote from dandl on October 29, 2019, 1:07 pmBut LINQ to SQL is not such a short step.Create the database, run a tool, generate a DBML, rinse, repeat (it's been a while). Updates use a side channel. And I don't think there is one for Java, just jOOQ? I haven't checked recently.
I vaguely recall other attempts, but I can't find them now. (Not that I've looked very thoroughly.) There may be cultural resistance in the Java community, which seems to be trending toward embracing SQL rather than wrapping it away to invisibility.
Quote from dandl on October 29, 2019, 1:07 pmThe RA has headspace -- a lot of people have been taught it, or can easily learn it. And a feature allowing experts to tune the generated SQL would give it another thing LINQ doesn't have.
But is there anything inherent in the RA that gives us expressive power -- or intuitiveness -- that LINQ or Streams doesn't have?
Yes. The trouble is that the strongest abstraction in the RA is exactly the opposite of the abstraction/intuition in most programmers' heads. Namely set-at-a-time thinking rather than Record-by-agonising-Record. SQL or LINQ or Streams seem to me low-level: they're a how-to-do not a what-to-do. SQL is at least a high enough level that an optimiser can figure the what-to-do (often) and turn your query into a different how-to-do. No doubt all the algorithms result in stream-based techniques ultimately; but the 'best' stream might not be the one the programmer asked for (for some value of 'best'). The worst is that they limit the programmer's thinking to 'which is to be my primary stream?'.
Consider a query like: list the Suppliers that supply all the Parts located in Paris. In a modern RA (I mean well beyond Codd's original operators, with a subset condition) you can express the 'big picture', and leave the thinking and the streaming to the execution plan. In SQL you need that convoluted
NOT EXISTS
double-negation (probably the optimiser can't see its way through that). In LINQ/Streams the programmer must set the stream (and likely they'll stream Suppliers, since that's what the start of the question asks for); and end up with obtuse/needlessly complex code.There's no explicit notion of projection per se but it isn't required in LINQ/Streams (there are alternatives); selection is equivalent; EXTEND is map (well, not really, but winds up being used that way); aggregation is fold or collect and various Collectors (in Java; I forget the equivalent in LINQ); set operators are there. JOIN has equivalents in LINQ but seems a bit chewy (but do-able) in Streams -- I'll have to revisit that -- and so on. In short, LINQ and Streams seem equivalent to RA in expressive power, just different.
Being able to tune generated SQL sounds good, but presumably isn't bound to an RA implementation.
The trouble with 'tuning' SQL from an implementor's point of view is that SQL's semantics is so pernickety and ad-hoc (and not the semantics of RA). So how do you get any sanity checking that your optimisation will give the same answer in all cases. (It's really hard to build up a test suite that will exercise all the obscure corners of SQL's semantics.) So not only is it not bound to an RA implementation, you don't get what you would usually rely on with RA.
Quote from Dave Voorhis on October 29, 2019, 2:06 pmQuote from dandl on October 29, 2019, 1:07 pmQuote from Dave Voorhis on October 29, 2019, 8:37 amQuote from dandl on October 29, 2019, 12:21 amWe talk about the Object Relational Mismatch, but TTM shows us the problem is deeper than that. TTM/D presents a perfectly plausible type system in two parts:
- A scalar type system (RM Pre 4) reasonably similar to and compatible with value types (but not OO types) in other languages
- A non-scalar type system (RM Pre 6 and 7) designed specifically for the convenience of the 'usual operators of the relational algebra' (RM Pre 18), and highly incompatible with just about every other language.
The solutions on offer are:
- Implement TTM/D, as an island language with no bridges to anywhere. Write applications in industrial D. [This is the thrust of D&D]
- Implement TTM/D as a hosted language. Write applications in a mix of HL and D. [This is Andl, or Rel]
- Ignore TTM, implement a simple type system and a hackneyed version of the RA in a unique language (SQL). Write applications in a mix of HL and SQL, with some kind of bridge between them. [This is ORM, xDBC, +/- stored procedures]
- (variant on 3): embed or generate SQL in HL [This is LINQ to SQL, jOOQ, etc].
- Implement a modestly powerful type system and a simplified version of the RA in a minimal language, embed in HL. [This was the proposal I described recently, based on domains and JSON.]
- Find a new compromise: the intersection of a rich type system and a 'logical equivalent of the relation algebra' that is compatible with existing languages and can manage data held locally or on remote servers. [A dream?]
- Find a new direction: some method other than a novel type system to embed a 'logical equivalent of the relation algebra' into existing languages.
It seems (1) is a dead-end. I've done (2) and it's not going anywhere that I can see. The world has gone for 3 and 4, but not without pain. I know how to do (5), and I think it has possibilities.
But (6) and (7) would perhaps be more interesting. Thoughts?
6 and 7 are indeed more interesting, but lead to an inevitable question: Why do we want RA?
5 sounds like 3 but with a potentially awkward abstraction layer around SQL. It doesn't sound appealing, at least not to me.
Quote from dandl on October 29, 2019, 1:07 pmBut LINQ to SQL is not such a short step.Create the database, run a tool, generate a DBML, rinse, repeat (it's been a while). Updates use a side channel. And I don't think there is one for Java, just jOOQ? I haven't checked recently.
I vaguely recall other attempts, but I can't find them now. (Not that I've looked very thoroughly.) There may be cultural resistance in the Java community, which seems to be trending toward embracing SQL rather than wrapping it away to invisibility.
Quote from dandl on October 29, 2019, 1:07 pmThe RA has headspace -- a lot of people have been taught it, or can easily learn it. And a feature allowing experts to tune the generated SQL would give it another thing LINQ doesn't have.
But is there anything inherent in the RA that gives us expressive power -- or intuitiveness -- that LINQ or Streams doesn't have?
Yes. The trouble is that the strongest abstraction in the RA is exactly the opposite of the abstraction/intuition in most programmers' heads. Namely set-at-a-time thinking rather than Record-by-agonising-Record. SQL or LINQ or Streams seem to me low-level: they're a how-to-do not a what-to-do. SQL is at least a high enough level that an optimiser can figure the what-to-do (often) and turn your query into a different how-to-do. No doubt all the algorithms result in stream-based techniques ultimately; but the 'best' stream might not be the one the programmer asked for (for some value of 'best'). The worst is that they limit the programmer's thinking to 'which is to be my primary stream?'.
Consider a query like: list the Suppliers that supply all the Parts located in Paris. In a modern RA (I mean well beyond Codd's original operators, with a subset condition) you can express the 'big picture', and leave the thinking and the streaming to the execution plan. In SQL you need that convoluted NOT EXISTS
double-negation (probably the optimiser can't see its way through that). In LINQ/Streams the programmer must set the stream (and likely they'll stream Suppliers, since that's what the start of the question asks for); and end up with obtuse/needlessly complex code.
There's no explicit notion of projection per se but it isn't required in LINQ/Streams (there are alternatives); selection is equivalent; EXTEND is map (well, not really, but winds up being used that way); aggregation is fold or collect and various Collectors (in Java; I forget the equivalent in LINQ); set operators are there. JOIN has equivalents in LINQ but seems a bit chewy (but do-able) in Streams -- I'll have to revisit that -- and so on. In short, LINQ and Streams seem equivalent to RA in expressive power, just different.
Being able to tune generated SQL sounds good, but presumably isn't bound to an RA implementation.
The trouble with 'tuning' SQL from an implementor's point of view is that SQL's semantics is so pernickety and ad-hoc (and not the semantics of RA). So how do you get any sanity checking that your optimisation will give the same answer in all cases. (It's really hard to build up a test suite that will exercise all the obscure corners of SQL's semantics.) So not only is it not bound to an RA implementation, you don't get what you would usually rely on with RA.
Quote from dandl on October 30, 2019, 1:35 pmQuote from johnwcowan on October 29, 2019, 12:32 pmQuote from dandl on October 29, 2019, 10:28 amThis is problematic. If you write
join(relA,relB)
what types are the the two arguments, to make that possible?Something on the lines of
Set<T extends Tuple>
. Introspection will be necessary to determine joinability.So when do you do that?
So how do you decide which scalar types to support?
Any serializable type. The serialization algorithm doesn't have to be the standard Java one.
That could work.
Not imperative. It's an expression language which generates function calls visible in Java, with parameters passed in. There is a specific complication around dealing with open expressions, which I mentioned earlier and needs more thought.
The one exception: it might be useful to allow local temporary variables as a factoring convenience. They don't actually store anything.
LGTM, including the variables. Being able to log things is also valuable.
I lean towards default generated nonsense type names, with the ability to provide a meaningful name.
I think that presumes that the RAJ compiler has access to all the .raj files in the repo simultaneously, otherwise it cannot be sure of using the same class for the same tuple type throughout. My ugly convention allows separate compilation.
Not sure it works like that. W don't want a minor edit in a RAJ file to result in a type name change which ripples through the code. Needs some thought.
The name is never really needed.
If the Java code is to have variables whose type is a tuple, it needs access to the type names.
I guess so. This is rather awkward where every expression returns a value of a different tuple type (as a Java class) with name that is either (a) unpredictable (b) predictable but changes if the expression is modified (c) has to be named by the developer. I'll probably go for the last.
The semantics don't matter. The query returns a collection; an update consumes a collection. The form of that collection is unimportant. Duplicates will be removed by the database.
I suppose that works. But the semantics is that of a set, and using a Set interface more tightly constrains the behavior of the Java code.
I would use a simple array or list (IList<T> in C#).
Quote from johnwcowan on October 29, 2019, 12:32 pmQuote from dandl on October 29, 2019, 10:28 amThis is problematic. If you write
join(relA,relB)
what types are the the two arguments, to make that possible?Something on the lines of
Set<T extends Tuple>
. Introspection will be necessary to determine joinability.So when do you do that?
So how do you decide which scalar types to support?
Any serializable type. The serialization algorithm doesn't have to be the standard Java one.
That could work.
Not imperative. It's an expression language which generates function calls visible in Java, with parameters passed in. There is a specific complication around dealing with open expressions, which I mentioned earlier and needs more thought.
The one exception: it might be useful to allow local temporary variables as a factoring convenience. They don't actually store anything.
LGTM, including the variables. Being able to log things is also valuable.
I lean towards default generated nonsense type names, with the ability to provide a meaningful name.
I think that presumes that the RAJ compiler has access to all the .raj files in the repo simultaneously, otherwise it cannot be sure of using the same class for the same tuple type throughout. My ugly convention allows separate compilation.
Not sure it works like that. W don't want a minor edit in a RAJ file to result in a type name change which ripples through the code. Needs some thought.
The name is never really needed.
If the Java code is to have variables whose type is a tuple, it needs access to the type names.
I guess so. This is rather awkward where every expression returns a value of a different tuple type (as a Java class) with name that is either (a) unpredictable (b) predictable but changes if the expression is modified (c) has to be named by the developer. I'll probably go for the last.
The semantics don't matter. The query returns a collection; an update consumes a collection. The form of that collection is unimportant. Duplicates will be removed by the database.
I suppose that works. But the semantics is that of a set, and using a Set interface more tightly constrains the behavior of the Java code.
I would use a simple array or list (IList<T> in C#).
Quote from dandl on October 30, 2019, 11:26 pmOne further thought: if this is to work, it should be easy to retrofit.
That is, it should be possible to find existing applications that conform to accepted development paradigms, and modify them to use a proposed RA-DSL product for a modest investment in effort in order to obtain an identifiable benefit. If that's true, and there is a reasonable implementation path, then it's worth a try. Otherwise not.
From what I know of application development in C# and Java, I would look for several possible features:
- self-contained, responsible for database creation and maintenance
- a clearly established data abstraction layer
- simple data transfer objects rather than a complex object model
- pervasive use of SQL fragments
- not deeply coupled to a specific API (such as an ORM).
[@Dave: have you published an application that fits the bill?]
The benefit should include one or more of:
- reduction in code volume
- bring the database start-up/maintenance process 'in-house'
- backend portability
- better factoring
- detect a few bugs along the way
- better testing.
But that's just guesswork, the only way to find out is to go read code.
One further thought: if this is to work, it should be easy to retrofit.
That is, it should be possible to find existing applications that conform to accepted development paradigms, and modify them to use a proposed RA-DSL product for a modest investment in effort in order to obtain an identifiable benefit. If that's true, and there is a reasonable implementation path, then it's worth a try. Otherwise not.
From what I know of application development in C# and Java, I would look for several possible features:
- self-contained, responsible for database creation and maintenance
- a clearly established data abstraction layer
- simple data transfer objects rather than a complex object model
- pervasive use of SQL fragments
- not deeply coupled to a specific API (such as an ORM).
[@Dave: have you published an application that fits the bill?]
The benefit should include one or more of:
- reduction in code volume
- bring the database start-up/maintenance process 'in-house'
- backend portability
- better factoring
- detect a few bugs along the way
- better testing.
But that's just guesswork, the only way to find out is to go read code.
Quote from Dave Voorhis on October 30, 2019, 11:46 pmQuote from dandl on October 30, 2019, 11:26 pmOne further thought: if this is to work, it should be easy to retrofit.
That is, it should be possible to find existing applications that conform to accepted development paradigms, and modify them to use a proposed RA-DSL product for a modest investment in effort in order to obtain an identifiable benefit. If that's true, and there is a reasonable implementation path, then it's worth a try. Otherwise not.
From what I know of application development in C# and Java, I would look for several possible features:
- self-contained, responsible for database creation and maintenance
- a clearly established data abstraction layer
- simple data transfer objects rather than a complex object model
- pervasive use of SQL fragments
- not deeply coupled to a specific API (such as an ORM).
[@Dave: have you published an application that fits the bill?]
I have just such an application -- a partially-finished but illustrative Java-based Web application (which would be finished except a change in policy & procedure meant it turned out to be unnecessary) -- but it isn't published anywhere. I've been intending to turn it into a demo of a simple Web application framework I built -- which I'm going to release as open source -- but I need to strip out some bits that are notionally proprietary to the client for whom it was built.
But it intentionally doesn't conform to generally "accepted development paradigms", though it ticks all of your boxes. It conforms to my accepted development paradigms and intentionally avoids the "standard" approaches. A better example would be to take a demo written using the usual Java Web application stack -- Hibernate and Spring -- and show how you can significantly improve on them.
Quote from dandl on October 30, 2019, 11:26 pmOne further thought: if this is to work, it should be easy to retrofit.
That is, it should be possible to find existing applications that conform to accepted development paradigms, and modify them to use a proposed RA-DSL product for a modest investment in effort in order to obtain an identifiable benefit. If that's true, and there is a reasonable implementation path, then it's worth a try. Otherwise not.
From what I know of application development in C# and Java, I would look for several possible features:
- self-contained, responsible for database creation and maintenance
- a clearly established data abstraction layer
- simple data transfer objects rather than a complex object model
- pervasive use of SQL fragments
- not deeply coupled to a specific API (such as an ORM).
[@Dave: have you published an application that fits the bill?]
I have just such an application -- a partially-finished but illustrative Java-based Web application (which would be finished except a change in policy & procedure meant it turned out to be unnecessary) -- but it isn't published anywhere. I've been intending to turn it into a demo of a simple Web application framework I built -- which I'm going to release as open source -- but I need to strip out some bits that are notionally proprietary to the client for whom it was built.
But it intentionally doesn't conform to generally "accepted development paradigms", though it ticks all of your boxes. It conforms to my accepted development paradigms and intentionally avoids the "standard" approaches. A better example would be to take a demo written using the usual Java Web application stack -- Hibernate and Spring -- and show how you can significantly improve on them.
Quote from dandl on October 31, 2019, 9:28 amQuote from Dave Voorhis on October 30, 2019, 11:46 pmQuote from dandl on October 30, 2019, 11:26 pmOne further thought: if this is to work, it should be easy to retrofit.
That is, it should be possible to find existing applications that conform to accepted development paradigms, and modify them to use a proposed RA-DSL product for a modest investment in effort in order to obtain an identifiable benefit. If that's true, and there is a reasonable implementation path, then it's worth a try. Otherwise not.
From what I know of application development in C# and Java, I would look for several possible features:
- self-contained, responsible for database creation and maintenance
- a clearly established data abstraction layer
- simple data transfer objects rather than a complex object model
- pervasive use of SQL fragments
- not deeply coupled to a specific API (such as an ORM).
[@Dave: have you published an application that fits the bill?]
I have just such an application -- a partially-finished but illustrative Java-based Web application (which would be finished except a change in policy & procedure meant it turned out to be unnecessary) -- but it isn't published anywhere. I've been intending to turn it into a demo of a simple Web application framework I built -- which I'm going to release as open source -- but I need to strip out some bits that are notionally proprietary to the client for whom it was built.
I look forward to seeing it. But I'm really not interested in targeting the Web, at least initially. There are my users: link. Look for the "Power User Gap". The ones who use Access because it sits (sat) just above Excel and well below the enterprise toolset. Yes, sooner or later they want Web but get the essentials right first. Database; UI with menus, forms and tables; reports with totalling; a bit of coding to glue it together. Why is that so hard?
But it intentionally doesn't conform to generally "accepted development paradigms", though it ticks all of your boxes. It conforms to my accepted development paradigms and intentionally avoids the "standard" approaches. A better example would be to take a demo written using the usual Java Web application stack -- Hibernate and Spring -- and show how you can significantly improve on them.
My paradigms do not extend to choosing specific ancillary products. I want to write code in a style that is familiar and readable, and not have to fight with other products to get simple stuff to work.
My aim would be to make both unnecessary for users who have no investment in either, and don't want any of the high end features. If you look at this and page down a bit, you will eventually find the application code.
public static void main(String[] args) { Resource resource = new FileSystemResource( "./src/resources/spring-hibernate.xml"); BeanFactory factory = new XmlBeanFactory(resource); Employee employee = new Employee(); employee.setId("123"); employee.setName("ABC"); employee.setAge(20); employee.setSalary(15000.00d); EmployeeDao employeeDao = (EmployeeDao)factory.getBean( "employeeDao"); employeeDao.saveOrUpdate(employee); Employee empResult = employeeDao.getEmployee("123"); System.out.println(empResult); }[This Wordpress editor is seriously bad.]
You ought to be able to write (or edit) code like this and not much else, and get it all working quickly and easily, from a standing start. It's been my goal for 30 years or more, but it doesn't seem to get any closer. Every generation of technology just makes it that much harder.
BTW we are using Knack for a club I belong to, and it really is deja vu all over again. Why is it so hard to do what looks so easy? Knack is the wrong way to go, in my view, but I have to admit the results are OK.
Quote from Dave Voorhis on October 30, 2019, 11:46 pmQuote from dandl on October 30, 2019, 11:26 pmOne further thought: if this is to work, it should be easy to retrofit.
That is, it should be possible to find existing applications that conform to accepted development paradigms, and modify them to use a proposed RA-DSL product for a modest investment in effort in order to obtain an identifiable benefit. If that's true, and there is a reasonable implementation path, then it's worth a try. Otherwise not.
From what I know of application development in C# and Java, I would look for several possible features:
- self-contained, responsible for database creation and maintenance
- a clearly established data abstraction layer
- simple data transfer objects rather than a complex object model
- pervasive use of SQL fragments
- not deeply coupled to a specific API (such as an ORM).
[@Dave: have you published an application that fits the bill?]
I have just such an application -- a partially-finished but illustrative Java-based Web application (which would be finished except a change in policy & procedure meant it turned out to be unnecessary) -- but it isn't published anywhere. I've been intending to turn it into a demo of a simple Web application framework I built -- which I'm going to release as open source -- but I need to strip out some bits that are notionally proprietary to the client for whom it was built.
I look forward to seeing it. But I'm really not interested in targeting the Web, at least initially. There are my users: link. Look for the "Power User Gap". The ones who use Access because it sits (sat) just above Excel and well below the enterprise toolset. Yes, sooner or later they want Web but get the essentials right first. Database; UI with menus, forms and tables; reports with totalling; a bit of coding to glue it together. Why is that so hard?
But it intentionally doesn't conform to generally "accepted development paradigms", though it ticks all of your boxes. It conforms to my accepted development paradigms and intentionally avoids the "standard" approaches. A better example would be to take a demo written using the usual Java Web application stack -- Hibernate and Spring -- and show how you can significantly improve on them.
My paradigms do not extend to choosing specific ancillary products. I want to write code in a style that is familiar and readable, and not have to fight with other products to get simple stuff to work.
My aim would be to make both unnecessary for users who have no investment in either, and don't want any of the high end features. If you look at this and page down a bit, you will eventually find the application code.
public static void main(String[] args) { Resource resource = new FileSystemResource( "./src/resources/spring-hibernate.xml"); BeanFactory factory = new XmlBeanFactory(resource); Employee employee = new Employee(); employee.setId("123"); employee.setName("ABC"); employee.setAge(20); employee.setSalary(15000.00d); EmployeeDao employeeDao = (EmployeeDao)factory.getBean( "employeeDao"); employeeDao.saveOrUpdate(employee); Employee empResult = employeeDao.getEmployee("123"); System.out.println(empResult); }
[This Wordpress editor is seriously bad.]
You ought to be able to write (or edit) code like this and not much else, and get it all working quickly and easily, from a standing start. It's been my goal for 30 years or more, but it doesn't seem to get any closer. Every generation of technology just makes it that much harder.
BTW we are using Knack for a club I belong to, and it really is deja vu all over again. Why is it so hard to do what looks so easy? Knack is the wrong way to go, in my view, but I have to admit the results are OK.
Quote from Dave Voorhis on October 31, 2019, 10:18 amQuote from dandl on October 31, 2019, 9:28 amQuote from Dave Voorhis on October 30, 2019, 11:46 pmQuote from dandl on October 30, 2019, 11:26 pmOne further thought: if this is to work, it should be easy to retrofit.
That is, it should be possible to find existing applications that conform to accepted development paradigms, and modify them to use a proposed RA-DSL product for a modest investment in effort in order to obtain an identifiable benefit. If that's true, and there is a reasonable implementation path, then it's worth a try. Otherwise not.
From what I know of application development in C# and Java, I would look for several possible features:
- self-contained, responsible for database creation and maintenance
- a clearly established data abstraction layer
- simple data transfer objects rather than a complex object model
- pervasive use of SQL fragments
- not deeply coupled to a specific API (such as an ORM).
[@Dave: have you published an application that fits the bill?]
I have just such an application -- a partially-finished but illustrative Java-based Web application (which would be finished except a change in policy & procedure meant it turned out to be unnecessary) -- but it isn't published anywhere. I've been intending to turn it into a demo of a simple Web application framework I built -- which I'm going to release as open source -- but I need to strip out some bits that are notionally proprietary to the client for whom it was built.
I look forward to seeing it. But I'm really not interested in targeting the Web, at least initially. There are my users: link. Look for the "Power User Gap". The ones who use Access because it sits (sat) just above Excel and well below the enterprise toolset. Yes, sooner or later they want Web but get the essentials right first. Database; UI with menus, forms and tables; reports with totalling; a bit of coding to glue it together. Why is that so hard?
If current progress continues, soon it won't be. It's what my datasheet project is quite rapidly becoming. I have these major components built or working works-in-progress, and largely independently:
- The Web framework I mentioned above, which -- along with a big bag of Web UI stuff that will get moved into #2 below -- includes a light abstraction layer on top of JDBC, plus database schema versioning tools to simplify deploying application updates, plus a built-in database schema (currently for PostgreSQL, but targeting others is trivial) and tools for handling user authentication and privileges.
- A cross-platform UI toolkit (consisting largely of a set of helper classes to abstract away differences between two Eclipse.org project UI widget kits, SWT for desktop and RAP/RWT for Web) that allows you to build applications for Windows, MacOS, Linux, and the Web with native look-and-feel on the desktop and a "desktop application" L&F for the Web -- all from one Java code base. It means the Web doesn't have to be treated specially; it's just one of the target platforms. Soon to be released on its own as an open source project called Spoing, which is a tongue-in-cheek riff on Spring (Java framework) and Swing (SWT stands for Swing Widget Toolkit.)
- A collection of grid-oriented UI tools (built using #2) for editing data, displaying query results, modifying database schemata, and quickly constructing forms (though you can also write them in Java or use Eclipse's WindowBuilder forms painter to generate Java.) This is a work-in-progress, and probably the least refined and complete of everything I've listed here.
- A toolkit for easily deploying Web and desktop Java applications as native applications with a bundled, stripped Java runtime for Windows, MacOS and Linux. It's currently used to deploy Rel, and it's what allowed me to improve startup times and reduce distribution size.
- Rel's graphical query editor, which will be adapted to work with Java's Streams API.
Basically, I need to seamlessly bolt all five together and significantly improve and extend #3. Obviously, it's a work in progress, but it's getting there. It's all straightforward, plain-old readable Java, too. No XML, no JSON, no hold-a-class-name-near-it-and-it-might-magically-pop-into-place dependency-injection frameworks, no annotation-based sublanguage to learn, no new obscure and un-debuggable DSLs.
I don't have anything for generating reports yet. I'm going to look at integrating BIRT.
Quote from dandl on October 31, 2019, 9:28 amQuote from Dave Voorhis on October 30, 2019, 11:46 pmQuote from dandl on October 30, 2019, 11:26 pmOne further thought: if this is to work, it should be easy to retrofit.
That is, it should be possible to find existing applications that conform to accepted development paradigms, and modify them to use a proposed RA-DSL product for a modest investment in effort in order to obtain an identifiable benefit. If that's true, and there is a reasonable implementation path, then it's worth a try. Otherwise not.
From what I know of application development in C# and Java, I would look for several possible features:
- self-contained, responsible for database creation and maintenance
- a clearly established data abstraction layer
- simple data transfer objects rather than a complex object model
- pervasive use of SQL fragments
- not deeply coupled to a specific API (such as an ORM).
[@Dave: have you published an application that fits the bill?]
I have just such an application -- a partially-finished but illustrative Java-based Web application (which would be finished except a change in policy & procedure meant it turned out to be unnecessary) -- but it isn't published anywhere. I've been intending to turn it into a demo of a simple Web application framework I built -- which I'm going to release as open source -- but I need to strip out some bits that are notionally proprietary to the client for whom it was built.
I look forward to seeing it. But I'm really not interested in targeting the Web, at least initially. There are my users: link. Look for the "Power User Gap". The ones who use Access because it sits (sat) just above Excel and well below the enterprise toolset. Yes, sooner or later they want Web but get the essentials right first. Database; UI with menus, forms and tables; reports with totalling; a bit of coding to glue it together. Why is that so hard?
If current progress continues, soon it won't be. It's what my datasheet project is quite rapidly becoming. I have these major components built or working works-in-progress, and largely independently:
- The Web framework I mentioned above, which -- along with a big bag of Web UI stuff that will get moved into #2 below -- includes a light abstraction layer on top of JDBC, plus database schema versioning tools to simplify deploying application updates, plus a built-in database schema (currently for PostgreSQL, but targeting others is trivial) and tools for handling user authentication and privileges.
- A cross-platform UI toolkit (consisting largely of a set of helper classes to abstract away differences between two Eclipse.org project UI widget kits, SWT for desktop and RAP/RWT for Web) that allows you to build applications for Windows, MacOS, Linux, and the Web with native look-and-feel on the desktop and a "desktop application" L&F for the Web -- all from one Java code base. It means the Web doesn't have to be treated specially; it's just one of the target platforms. Soon to be released on its own as an open source project called Spoing, which is a tongue-in-cheek riff on Spring (Java framework) and Swing (SWT stands for Swing Widget Toolkit.)
- A collection of grid-oriented UI tools (built using #2) for editing data, displaying query results, modifying database schemata, and quickly constructing forms (though you can also write them in Java or use Eclipse's WindowBuilder forms painter to generate Java.) This is a work-in-progress, and probably the least refined and complete of everything I've listed here.
- A toolkit for easily deploying Web and desktop Java applications as native applications with a bundled, stripped Java runtime for Windows, MacOS and Linux. It's currently used to deploy Rel, and it's what allowed me to improve startup times and reduce distribution size.
- Rel's graphical query editor, which will be adapted to work with Java's Streams API.
Basically, I need to seamlessly bolt all five together and significantly improve and extend #3. Obviously, it's a work in progress, but it's getting there. It's all straightforward, plain-old readable Java, too. No XML, no JSON, no hold-a-class-name-near-it-and-it-might-magically-pop-into-place dependency-injection frameworks, no annotation-based sublanguage to learn, no new obscure and un-debuggable DSLs.
I don't have anything for generating reports yet. I'm going to look at integrating BIRT.
Quote from dandl on October 31, 2019, 1:10 pmQuote from Dave Voorhis on October 31, 2019, 10:18 amI look forward to seeing it. But I'm really not interested in targeting the Web, at least initially. There are my users: link. Look for the "Power User Gap". The ones who use Access because it sits (sat) just above Excel and well below the enterprise toolset. Yes, sooner or later they want Web but get the essentials right first. Database; UI with menus, forms and tables; reports with totalling; a bit of coding to glue it together. Why is that so hard?
If current progress continues, soon it won't be. It's what my datasheet project is quite rapidly becoming. I have these major components built or working works-in-progress, and largely independently:
- The Web framework I mentioned above, which -- along with a big bag of Web UI stuff that will get moved into #2 below -- includes a light abstraction layer on top of JDBC, plus database schema versioning tools to simplify deploying application updates, plus a built-in database schema (currently for PostgreSQL, but targeting others is trivial) and tools for handling user authentication and privileges.
The database stuff I understand, but didn't you miss chunk of Web stuff? What serves pages?
- A cross-platform UI toolkit (consisting largely of a set of helper classes to abstract away differences between two Eclipse.org project UI widget kits, SWT for desktop and RAP/RWT for Web) that allows you to build applications for Windows, MacOS, Linux, and the Web with native look-and-feel on the desktop and a "desktop application" L&F for the Web -- all from one Java code base. It means the Web doesn't have to be treated specially; it's just one of the target platforms. Soon to be released on its own as an open source project called Spoing, which is a tongue-in-cheek riff on Spring (Java framework) and Swing (SWT stands for Swing Widget Toolkit.)
If it works, wonderful. What I'm seeing in the wild is JS frameworks (React, Vue, etc), which kind of pretend the desktop is a dinosaur, best forgotten. Where can I find something to read?
- A collection of grid-oriented UI tools (built using #2) for editing data, displaying query results, modifying database schemata, and quickly constructing forms (though you can also write them in Java or use Eclipse's WindowBuilder forms painter to generate Java.) This is a work-in-progress, and probably the least refined and complete of everything I've listed here.
- A toolkit for easily deploying Web and desktop Java applications as native applications with a bundled, stripped Java runtime for Windows, MacOS and Linux. It's currently used to deploy Rel, and it's what allowed me to improve startup times and reduce distribution size.
- Rel's graphical query editor, which will be adapted to work with Java's Streams API.
Basically, I need to seamlessly bolt all five together and significantly improve and extend #3. Obviously, it's a work in progress, but it's getting there. It's all straightforward, plain-old readable Java, too. No XML, no JSON, no hold-a-class-name-near-it-and-it-might-magically-pop-into-place dependency-injection frameworks, no annotation-based sublanguage to learn, no new obscure and un-debuggable DSLs.
Sounds like your shopping list is a lot like mine.Yes, for some strange reason I've never seen an IOC/DI framework that I didn't dislike. In principle, a suitably designed DSL/Python will beat XML for gluing stuff together.
I look forward to seeing a pre-alpha.
I don't have anything for generating reports yet. I'm going to look at integrating BIRT.
Looks promising. Supports POJOs, so it should integrate. XML again, but at least just as a persistence mechanism, not as a pseudo-programming language.
Quote from Dave Voorhis on October 31, 2019, 10:18 amI look forward to seeing it. But I'm really not interested in targeting the Web, at least initially. There are my users: link. Look for the "Power User Gap". The ones who use Access because it sits (sat) just above Excel and well below the enterprise toolset. Yes, sooner or later they want Web but get the essentials right first. Database; UI with menus, forms and tables; reports with totalling; a bit of coding to glue it together. Why is that so hard?
If current progress continues, soon it won't be. It's what my datasheet project is quite rapidly becoming. I have these major components built or working works-in-progress, and largely independently:
- The Web framework I mentioned above, which -- along with a big bag of Web UI stuff that will get moved into #2 below -- includes a light abstraction layer on top of JDBC, plus database schema versioning tools to simplify deploying application updates, plus a built-in database schema (currently for PostgreSQL, but targeting others is trivial) and tools for handling user authentication and privileges.
The database stuff I understand, but didn't you miss chunk of Web stuff? What serves pages?
- A cross-platform UI toolkit (consisting largely of a set of helper classes to abstract away differences between two Eclipse.org project UI widget kits, SWT for desktop and RAP/RWT for Web) that allows you to build applications for Windows, MacOS, Linux, and the Web with native look-and-feel on the desktop and a "desktop application" L&F for the Web -- all from one Java code base. It means the Web doesn't have to be treated specially; it's just one of the target platforms. Soon to be released on its own as an open source project called Spoing, which is a tongue-in-cheek riff on Spring (Java framework) and Swing (SWT stands for Swing Widget Toolkit.)
If it works, wonderful. What I'm seeing in the wild is JS frameworks (React, Vue, etc), which kind of pretend the desktop is a dinosaur, best forgotten. Where can I find something to read?
- A collection of grid-oriented UI tools (built using #2) for editing data, displaying query results, modifying database schemata, and quickly constructing forms (though you can also write them in Java or use Eclipse's WindowBuilder forms painter to generate Java.) This is a work-in-progress, and probably the least refined and complete of everything I've listed here.
- A toolkit for easily deploying Web and desktop Java applications as native applications with a bundled, stripped Java runtime for Windows, MacOS and Linux. It's currently used to deploy Rel, and it's what allowed me to improve startup times and reduce distribution size.
- Rel's graphical query editor, which will be adapted to work with Java's Streams API.
Basically, I need to seamlessly bolt all five together and significantly improve and extend #3. Obviously, it's a work in progress, but it's getting there. It's all straightforward, plain-old readable Java, too. No XML, no JSON, no hold-a-class-name-near-it-and-it-might-magically-pop-into-place dependency-injection frameworks, no annotation-based sublanguage to learn, no new obscure and un-debuggable DSLs.
Sounds like your shopping list is a lot like mine.Yes, for some strange reason I've never seen an IOC/DI framework that I didn't dislike. In principle, a suitably designed DSL/Python will beat XML for gluing stuff together.
I look forward to seeing a pre-alpha.
I don't have anything for generating reports yet. I'm going to look at integrating BIRT.
Looks promising. Supports POJOs, so it should integrate. XML again, but at least just as a persistence mechanism, not as a pseudo-programming language.