The Third Manifesto ForumThe Third Manifesto: TTM General discussionLife after D

Please or Register to create posts and topics.

Life after D

#11 · March 16, 2021, 6:22 pm

Quote from Dave Voorhis on March 16, 2021, 6:22 pm

Quote from dandl on March 16, 2021, 12:20 am

APL is highest of all, so I'm not sure that metric means much.

I subscribe to the principle: that every major programming project generates its own language, usually a bad one. Otherwise known as LYHINLYN: the language you have is not the language you need (to solve this problem). So first design the TLYN, implement that using TLYH, then solve the problem in your new language. I read that somewhere, but I have no idea where and I can't find it.

Lisp subscribes heavily to that principle. Alan Kay said: "Lisp is a building material, not a programming language".

Notably, Lisp is one of several language families that supports -- and some encourage -- using to create domain-specific languages. Whether syntactically-distinct DSLs are bad, good or indifferent is a matter of some debate (and perhaps personal taste), but it's worth noting that virtually every library of routines or class library is effectively a DSL. They share the syntax of the host language, obviously, but semantically that's what a library is.

Therefore, if you feel the urge to create a syntactically-distinct DSL, the question that comes to mind is what is it about the host language that you don't like?

I'd prefer a seamless, integrated language that supports all the abstraction, expressiveness, and shortcuts that a professional programmer might want, but at the same time supports simple, intuitive, easy syntax for non-professional-programmer developers, so that DSL capability is provided not by defining new syntactically-distinct languages, but by defining libraries that anyone can use.

I don't think these wishes -- for abstract power that professional programmers will like, along with ease of use for beginners and non-professionals -- are contradictory or unreasonable.

Python seems part way along that path. I always thought its object orientation was a stumbling block to achieving it, because its OO support is ungainly, half-baked and awkward. It turns out that most non-professional Python programmers never define their own classes, so that's an issue that doesn't come up. But for the beginner/non-professional who becomes proficient enough to want to use classes, they are there.

I imagine the new language we're talking about being the same, but ideally without the flaws. There can be features that the beginner/non-professional might never use, but they are there in case they're wanted. The language should thus support a continuous spectrum of developers -- from rank beginner non-programmers at one end to fully-capable professionals at the other -- with none of the discontinuous breaks in assumed developer ability (or target language) that are inevitably be created by defining a 4GL with "escapes" to (a) 3GL(s).

What I'm on about is separating tool building from tool use: making it way easier to create, maintain and extend the LYN. APL is a super example of the LYN for writing certain kinds of mathematical expressions, hampered by some awkward choices of symbols. We should be able to create a 'New APL' easily to replace it, but we can't.

This is mostly an aside, but there is a 'New APL'. It's called J, designed and implemented in part by the same Ken Iverson who invented APL.

From jsoftware.com: "J is a high-level, general-purpose programming language that is particularly suited to the mathematical, statistical, and logical analysis of data. [...] Jd ( J database) is a high-performance columnar RDBMS written in J that is geared toward storing and analyzing large amounts of data. Jd is free for non-commercial use. Jd lives openly and dynamically in the J execution and development environment, so that the full power of J is available to the application developer. For example, Jd columns are mapped to J nouns, so built-in J primitives can apply directly to the data."

Sounds like it might be just what you're looking for.

Ditto for 4GLs, like dBase and Powerflex.

I don't know about Powerflex or Dataflex (which I understand it resembles?) but in the 1980's I created some utilities (in assembly language -- they were Terminate-and-Stay-Resident utilities, back when those were a thing) for a consultant who used Dataflex.

I think that highlights one of the limitations of such languages, which is that its developers inevitably want functionality that they'd like to have been able to create within the language, but couldn't.

(Well, maybe not TSR utilities, as those were a special category of hack, but the principle is good -- the language should be capable, within reason, of doing whatever a developer might want even if the average developer can't.)

As for dBase, it was barely a 4GL in the usual sense, except for having query and form primitives (sort of, I guess.) It was, in every respect, an abomination.
IMO 4GL was too early, it was solving the wrong problem for the time, the priority was 3GL. But the whole 3GL thing is now good enough, there is nothing important to do, just tweaks. It's time to go back and do 4GL again: less is more.

shorter: less lines of code (to do common repetitive programming)

safer: less room for bugs

higher abstraction: higher level programmable objects (forms, datasets, collections, not glorified machine words, values, pointers)
<snip>
Heavens. 1987 called and wants its UI definition language back.
Certainly not. The point of this is to illustrate those 3 features: shorter, safer, higher.

I think a more modern approach would be to define the interface in HTML/CSS, and have a tool that parses out the forms so you can use them from code.

Why ever would that be a good idea? Those are the assembly languages of the modern browser, not even one step up from rock bottom. Horrible.
I've seen it done effectively. The UX/UI people produce the front-end using the tools they know (HTML/CSS or whatever design tool emits it) to produce the look-and-feel they want, and the coding tool slurps up all the form field definitions and fill-in-able HTML elements (of some class, I guess? I didn't look) and makes them available to the code.

The UX/UI folks aren't forced to learn new stuff to implement designs or be relegated to using wireframes or creating non-production HTML/CSS, and the programmers aren't forced to laboriously re-implement designs in HTML/CSS/code provided by UX/UI folks who have been relegated to using wireframe tools because they can't write production HTML/CSS.

Of course, it presumes the goal is custom UI/UX designs for end-user Web sites. If it's only manual data entry that matters and look-and-feel isn't a concern, then any form painting / coding tool will do.
None of this has anything whatever to do with the issue of 'Life after D'. The mere fact that you even raise those as issues of concern says we're a long way off a solution.

Yes, use tools to make visual imagery that looks as desired, but no, never look at the save file format. Do you routinely read the SVG emitted by Inkscape or the FBX emitted by Autodesk tools or the PS emitted by Adobe? Of course not!We assume a save file form for UI/UX tools and a save file format for data model design, and we don't poke around inside: those are just UI objects and dataset objects. The question is what language we use to manipulate those objects. And the aim is: sweet spot at 100 lines of code, but absolutely no program over 300 LOC.
Sorry, you've lost me here. We must be talking past each other.
I mean:

Use visual tools to create visual imagery (the saved file format is something standardised and interoperable)

Use programming languages to manipulate those visual images as opaque objects.
I don't see how that relates to HTML/CSS, and the fact that UI/UX designers comfortably work in and with it.

Not at this point. I care about tools created by tool creators to be used by tool users. A lumberjack is an expert in felling trees, but someone has to make axes. I am a tool maker, my customers are programmers who use my tools. What language tool should I make for them to achieve shorter, safer, higher?

If it's the pseudo-technical programmers I work with, it's probably a nice library -- or set of libraries -- for Python.

Of course, there's a possibility here that your pseudo-technical programmers are very different from mine. I am reminded of the parable of the blind men and an elephant here, and I'm aware that despite working in notionally the same field, our experience of it could be almost completely different. It only takes a few distinct client niches to create a wholly different experience of what typical IT is, and what its developers do, expect, want, need, or use.

Yes, the main (if not the only) 'shorter-safer-higher' we have right now is the library of classes and functions, but that still leaves the accidental complexity of the programming task fully exposed. Have you written any C++ lately? We talk about code reuse, components and modules but that's really hard to do with the tools we have. But that still leaves the use of the tool in the same language used to create the tool, and that's a mistake: the language is complex, powerful and unsafe in the wrong hands. That's taking the fully-equipped machine shop everywhere when all you want to do is change a light bulb or hand a picture.

So I'm advocating languages that can expose a controlled subset as a new LYN. If you take my 4GL as an example, you can expose a LYN that does that data entry task in the same few lines of code. Once you discover a need for a feature not included, the LYN manager extends the language by exposing features as needed.

Actually, yes, I have written C++ recently and it was not a particularly delightful experience.

I don't think we have to go to C++ levels of complexity to achieve a reasonable, broad spectrum of language usability. Indeed, I think Python, Ruby, even Julia are perhaps much of the way there.

4GLs wind up going to the opposite extreme, such that C++ and 4GLs represent opposite extremes: The former is a language too complex to be friendly, and the latter is invariably a language too restricted to be friendly.

"LYN manager"???

Again why? Every language creates data structures and every data structure is filling in checkboxes. How do we add new checkboxes as the need arises?

That's where the strong 3GL comes in.

Only if it's used to build the LYN.

Finally, your interpretation of "abstraction" is odd. Compared to the current popular programming languages, higher levels of abstraction are (for example) declarative programming, pattern-matching (-based dispatch), meta-programming, generative programming, inferential systems, constraint-based programming, etc., and are always in comparison to lower levels. Prewritten libraries for specific uses such as forms are typically not abstractions in the programming sense, they're simply someone else writing some non-abstract code for you.

Notably, levels of abstraction are hierarchical, the way "shape" is an abstraction of "cube", "cube" is an abstraction of "building", "building" is an abstraction of "house", etc. What you're describing is not higher level abstraction but predefinition. E.g., what is "form" an abstraction of? (If you're thinking "a bunch of fields", that's composition rather than abstraction.)

I think you're quibbling over a choice of words. An older 3GL language let me think about strings, a newer one lets me think about collections of strings, now I want to think about documents or pages or forms. I know there will be bits and bytes all the way down to the metal, but I want languages that aim 'higher', however you define it.

Is it really a language you want?

Or better/suitable pre-built libraries for the languages you've got?

No, not libraries as now. The LYN has no accidental complexity: every line of code is there to do a job that is part of the end result, not dictated by the programming environment.

Accidental -- or unavoidable complexity, often scaffolding that's necessary in some context but mere overhead in another -- is a flaw of general-purpose language design. The solution is better general-purpose language design, not domain-specific 4GLs that replace accidental complexity with arduous extendibility, limited capability except outside of a narrow domain, and (inevitably) frustrating limitations or discontinuities (you need to escape to a 3GL here) as the user/programmer grows in ability and desire for non-predefined capability.

I know exactly what a regex can do, and I hate deeply the mental gymnastics I have go through to do it. I routinely do simple file system tasks well within the capabilities of Windows cmd, Powershell or bash and I routinely fail to get them right first time. I have a bunch of CSV files that comprise a data model, and there is hours of work before I can get the data to the point I can do SQL queries on it. None of these problems are handled by 3GLs per se, but all of them could be handled by tools implemented in a 3GL. We're not good enough at crating those tools.

Are you sure it's not simply that you are slow at using the tools we already have?

Of course I am, because the tools are crap: the knowledge load is too high, I can't know everything. The LYN I need is one to solve basic regex but I have to use the whole machine shop to do the job.

Except the tools aren't crap, because there are people who happily use them very successfully. Beware of mistaking your lack of experience with a tool for a lack of tool capability!

Far better, I think, to recognise where we can build on what people like about existing tools to make better tools.

We've gone the wrong way if we start by assuming that the tools others think are good are actually bad.

I don't mean that as an insult or deprecation, simply that your strengths -- and experience, and practice -- lie elsewhere. Nikki, my other half, can wire up Excel spreadsheets to do complex data crunching things in seconds and has whole infrastructure built in minutes and connected to external data stores whilst I'm still trying to figure out how to format a number in cell A1.

So the LYN for that job would capture her knowledge and let you (and me) perform at that same level.

I watched a demo a month ago by a colleague using Python and Pandas and Jupyter, and in four minutes (time-constrained demo sessions) he shlorped up data from some a weather Web site, did some nifty stats on it, and created a dynamic graph of the data.

I've done all those things and yes, that's a perfectly credible start of a LYN (although I did resent how long it took to install Jupyter).

Of course, they do that sort of thing all day, so they're quick at it. I can do something distantly similar with Rel for crunching through tabular data in CSV files and spreadsheets, but I wouldn't expect anyone else to do it (except Hugh.)

Likewise for other people and other tools.

But the key ingredient in every case is a lot of practice. If you're expecting some tool to suit your preconceptions so you can be immediately not just productive, but fast, with no prior practice, you may be looking for a chimera unless you create it all (perhaps including the underlying language) yourself.

No, I specifically want language tools that allow expert knowledge to be captured and exposed as the LYN for simpler folk to use. The 3GLs we have are enough (give or take a few tweaks), for the users who put in the 10,000 hours of practice, but the tools we have for creating a new LYN are poor. That's where I think the focus should be.

Again, this completes misses the key distinction between the roles of tool creator and tool user. The 4GL is a tool created for people with business needs who think they're programmers by people who really are programmers but find business stuff boring. You are an oddity: academic, compiler writer, but also application programmer, so perhaps you don't see this. In my experience only 1 in 10 programmers can write tools, and the other 9 really don't want to. The 1 in 10 really want all the stuff you've been banging on about, but the other 9 just want to churn out working code with the least possible effort. You won't solve this problem by designing esoteric languages exclusively for the high fliers: you need ways to let them efficiently build tools for everyone else. That's what a 4GL really is (or might have been): a way to divide tool builders from tool users.

I don't think I'm describing anything esoteric. I was actually thinking more along the lines of a better Python (in concept, not necessarily in syntax), but with greater ease and safety, and friendly access to higher-order capability and abstraction. If we were just going for esoteric, high-flying purity, we'd just pick Haskell or OCaml or whatever and be done with it.

But I'm thinking general-purpose abstraction rather than domain-specific predefinition. Specificity should be a feature of libraries written using the language, not features wired into the language.

Wiring features into languages is where 4GLs went wrong, and where they are still wrong. 4GLs invariably reflect a certain arrogance on the part of the 4GL vendor, that its users will only ever want to do things the way the 4GL wants them done. In reality, that almost invariably flies against what the developer wants to do -- even for developers who have not the slightest interest in anything but solving business problems.

That's why 4GLs are all but gone, 3GLs are pervasive, and everyone and their dog thinks learning Python is worthwhile.

You're getting closer. Python is still a crap language, but it makes great glue. Still way too much accidental complexity, but it's a step in the right direction.

Yes, 4GLs phase 1 are gone, but that was just a warmup. My LYN:

has zero accidental complexity

fully expresses the things that need to be done for business purpose

is maintainable and extensible as needed

maps readily unto the underlying 3GL(s)

achieves shorter, safer, higher.

Isn't that just a 'new' 4GL by another name?

Maybe, but I'm not convinced that "maps readily unto the underlying 3GL" is anywhere near as good as "3GL with zero accidental complexity, that can fully express the things that need to be done for business purpose via syntactically-friendly libraries, is maintainable and extensible as needed, and achieves what the user likes."

"What the user likes" is a (maybe the) key ingredient, and it isn't necessarily shorter, safer (sigh -- I have many debates with proponents of dynamic typing), or higher. It might include (or be) other qualities -- like fun to use -- that are much more persuasive in getting users to switch from whatever they use now.

Quote from dandl on March 16, 2021, 12:20 am

APL is highest of all, so I'm not sure that metric means much.

I subscribe to the principle: that every major programming project generates its own language, usually a bad one. Otherwise known as LYHINLYN: the language you have is not the language you need (to solve this problem). So first design the TLYN, implement that using TLYH, then solve the problem in your new language. I read that somewhere, but I have no idea where and I can't find it.

Lisp subscribes heavily to that principle. Alan Kay said: "Lisp is a building material, not a programming language".

Notably, Lisp is one of several language families that supports -- and some encourage -- using to create domain-specific languages. Whether syntactically-distinct DSLs are bad, good or indifferent is a matter of some debate (and perhaps personal taste), but it's worth noting that virtually every library of routines or class library is effectively a DSL. They share the syntax of the host language, obviously, but semantically that's what a library is.

Therefore, if you feel the urge to create a syntactically-distinct DSL, the question that comes to mind is what is it about the host language that you don't like?

I'd prefer a seamless, integrated language that supports all the abstraction, expressiveness, and shortcuts that a professional programmer might want, but at the same time supports simple, intuitive, easy syntax for non-professional-programmer developers, so that DSL capability is provided not by defining new syntactically-distinct languages, but by defining libraries that anyone can use.

I don't think these wishes -- for abstract power that professional programmers will like, along with ease of use for beginners and non-professionals -- are contradictory or unreasonable.

Python seems part way along that path. I always thought its object orientation was a stumbling block to achieving it, because its OO support is ungainly, half-baked and awkward. It turns out that most non-professional Python programmers never define their own classes, so that's an issue that doesn't come up. But for the beginner/non-professional who becomes proficient enough to want to use classes, they are there.

I imagine the new language we're talking about being the same, but ideally without the flaws. There can be features that the beginner/non-professional might never use, but they are there in case they're wanted. The language should thus support a continuous spectrum of developers -- from rank beginner non-programmers at one end to fully-capable professionals at the other -- with none of the discontinuous breaks in assumed developer ability (or target language) that are inevitably be created by defining a 4GL with "escapes" to (a) 3GL(s).

What I'm on about is separating tool building from tool use: making it way easier to create, maintain and extend the LYN. APL is a super example of the LYN for writing certain kinds of mathematical expressions, hampered by some awkward choices of symbols. We should be able to create a 'New APL' easily to replace it, but we can't.

This is mostly an aside, but there is a 'New APL'. It's called J, designed and implemented in part by the same Ken Iverson who invented APL.

From jsoftware.com: "J is a high-level, general-purpose programming language that is particularly suited to the mathematical, statistical, and logical analysis of data. [...] Jd ( J database) is a high-performance columnar RDBMS written in J that is geared toward storing and analyzing large amounts of data. Jd is free for non-commercial use. Jd lives openly and dynamically in the J execution and development environment, so that the full power of J is available to the application developer. For example, Jd columns are mapped to J nouns, so built-in J primitives can apply directly to the data."

Sounds like it might be just what you're looking for.

Ditto for 4GLs, like dBase and Powerflex.

I don't know about Powerflex or Dataflex (which I understand it resembles?) but in the 1980's I created some utilities (in assembly language -- they were Terminate-and-Stay-Resident utilities, back when those were a thing) for a consultant who used Dataflex.

I think that highlights one of the limitations of such languages, which is that its developers inevitably want functionality that they'd like to have been able to create within the language, but couldn't.

(Well, maybe not TSR utilities, as those were a special category of hack, but the principle is good -- the language should be capable, within reason, of doing whatever a developer might want even if the average developer can't.)

As for dBase, it was barely a 4GL in the usual sense, except for having query and form primitives (sort of, I guess.) It was, in every respect, an abomination.

IMO 4GL was too early, it was solving the wrong problem for the time, the priority was 3GL. But the whole 3GL thing is now good enough, there is nothing important to do, just tweaks. It's time to go back and do 4GL again: less is more.

shorter: less lines of code (to do common repetitive programming)

safer: less room for bugs

higher abstraction: higher level programmable objects (forms, datasets, collections, not glorified machine words, values, pointers)
<snip>
Heavens. 1987 called and wants its UI definition language back.
Certainly not. The point of this is to illustrate those 3 features: shorter, safer, higher.

I think a more modern approach would be to define the interface in HTML/CSS, and have a tool that parses out the forms so you can use them from code.

Why ever would that be a good idea? Those are the assembly languages of the modern browser, not even one step up from rock bottom. Horrible.
I've seen it done effectively. The UX/UI people produce the front-end using the tools they know (HTML/CSS or whatever design tool emits it) to produce the look-and-feel they want, and the coding tool slurps up all the form field definitions and fill-in-able HTML elements (of some class, I guess? I didn't look) and makes them available to the code.

The UX/UI folks aren't forced to learn new stuff to implement designs or be relegated to using wireframes or creating non-production HTML/CSS, and the programmers aren't forced to laboriously re-implement designs in HTML/CSS/code provided by UX/UI folks who have been relegated to using wireframe tools because they can't write production HTML/CSS.

Of course, it presumes the goal is custom UI/UX designs for end-user Web sites. If it's only manual data entry that matters and look-and-feel isn't a concern, then any form painting / coding tool will do.
None of this has anything whatever to do with the issue of 'Life after D'. The mere fact that you even raise those as issues of concern says we're a long way off a solution.

Yes, use tools to make visual imagery that looks as desired, but no, never look at the save file format. Do you routinely read the SVG emitted by Inkscape or the FBX emitted by Autodesk tools or the PS emitted by Adobe? Of course not!We assume a save file form for UI/UX tools and a save file format for data model design, and we don't poke around inside: those are just UI objects and dataset objects. The question is what language we use to manipulate those objects. And the aim is: sweet spot at 100 lines of code, but absolutely no program over 300 LOC.
Sorry, you've lost me here. We must be talking past each other.
I mean:

Use visual tools to create visual imagery (the saved file format is something standardised and interoperable)

Use programming languages to manipulate those visual images as opaque objects.

I don't see how that relates to HTML/CSS, and the fact that UI/UX designers comfortably work in and with it.

Not at this point. I care about tools created by tool creators to be used by tool users. A lumberjack is an expert in felling trees, but someone has to make axes. I am a tool maker, my customers are programmers who use my tools. What language tool should I make for them to achieve shorter, safer, higher?

If it's the pseudo-technical programmers I work with, it's probably a nice library -- or set of libraries -- for Python.

Of course, there's a possibility here that your pseudo-technical programmers are very different from mine. I am reminded of the parable of the blind men and an elephant here, and I'm aware that despite working in notionally the same field, our experience of it could be almost completely different. It only takes a few distinct client niches to create a wholly different experience of what typical IT is, and what its developers do, expect, want, need, or use.

Yes, the main (if not the only) 'shorter-safer-higher' we have right now is the library of classes and functions, but that still leaves the accidental complexity of the programming task fully exposed. Have you written any C++ lately? We talk about code reuse, components and modules but that's really hard to do with the tools we have. But that still leaves the use of the tool in the same language used to create the tool, and that's a mistake: the language is complex, powerful and unsafe in the wrong hands. That's taking the fully-equipped machine shop everywhere when all you want to do is change a light bulb or hand a picture.

So I'm advocating languages that can expose a controlled subset as a new LYN. If you take my 4GL as an example, you can expose a LYN that does that data entry task in the same few lines of code. Once you discover a need for a feature not included, the LYN manager extends the language by exposing features as needed.

Actually, yes, I have written C++ recently and it was not a particularly delightful experience.

I don't think we have to go to C++ levels of complexity to achieve a reasonable, broad spectrum of language usability. Indeed, I think Python, Ruby, even Julia are perhaps much of the way there.

4GLs wind up going to the opposite extreme, such that C++ and 4GLs represent opposite extremes: The former is a language too complex to be friendly, and the latter is invariably a language too restricted to be friendly.

"LYN manager"???

Again why? Every language creates data structures and every data structure is filling in checkboxes. How do we add new checkboxes as the need arises?

That's where the strong 3GL comes in.

Only if it's used to build the LYN.

Finally, your interpretation of "abstraction" is odd. Compared to the current popular programming languages, higher levels of abstraction are (for example) declarative programming, pattern-matching (-based dispatch), meta-programming, generative programming, inferential systems, constraint-based programming, etc., and are always in comparison to lower levels. Prewritten libraries for specific uses such as forms are typically not abstractions in the programming sense, they're simply someone else writing some non-abstract code for you.

Notably, levels of abstraction are hierarchical, the way "shape" is an abstraction of "cube", "cube" is an abstraction of "building", "building" is an abstraction of "house", etc. What you're describing is not higher level abstraction but predefinition. E.g., what is "form" an abstraction of? (If you're thinking "a bunch of fields", that's composition rather than abstraction.)

I think you're quibbling over a choice of words. An older 3GL language let me think about strings, a newer one lets me think about collections of strings, now I want to think about documents or pages or forms. I know there will be bits and bytes all the way down to the metal, but I want languages that aim 'higher', however you define it.

Is it really a language you want?

Or better/suitable pre-built libraries for the languages you've got?

No, not libraries as now. The LYN has no accidental complexity: every line of code is there to do a job that is part of the end result, not dictated by the programming environment.

Accidental -- or unavoidable complexity, often scaffolding that's necessary in some context but mere overhead in another -- is a flaw of general-purpose language design. The solution is better general-purpose language design, not domain-specific 4GLs that replace accidental complexity with arduous extendibility, limited capability except outside of a narrow domain, and (inevitably) frustrating limitations or discontinuities (you need to escape to a 3GL here) as the user/programmer grows in ability and desire for non-predefined capability.

I know exactly what a regex can do, and I hate deeply the mental gymnastics I have go through to do it. I routinely do simple file system tasks well within the capabilities of Windows cmd, Powershell or bash and I routinely fail to get them right first time. I have a bunch of CSV files that comprise a data model, and there is hours of work before I can get the data to the point I can do SQL queries on it. None of these problems are handled by 3GLs per se, but all of them could be handled by tools implemented in a 3GL. We're not good enough at crating those tools.

Are you sure it's not simply that you are slow at using the tools we already have?

Of course I am, because the tools are crap: the knowledge load is too high, I can't know everything. The LYN I need is one to solve basic regex but I have to use the whole machine shop to do the job.

Except the tools aren't crap, because there are people who happily use them very successfully. Beware of mistaking your lack of experience with a tool for a lack of tool capability!

Far better, I think, to recognise where we can build on what people like about existing tools to make better tools.

We've gone the wrong way if we start by assuming that the tools others think are good are actually bad.

I don't mean that as an insult or deprecation, simply that your strengths -- and experience, and practice -- lie elsewhere. Nikki, my other half, can wire up Excel spreadsheets to do complex data crunching things in seconds and has whole infrastructure built in minutes and connected to external data stores whilst I'm still trying to figure out how to format a number in cell A1.

So the LYN for that job would capture her knowledge and let you (and me) perform at that same level.

I watched a demo a month ago by a colleague using Python and Pandas and Jupyter, and in four minutes (time-constrained demo sessions) he shlorped up data from some a weather Web site, did some nifty stats on it, and created a dynamic graph of the data.

I've done all those things and yes, that's a perfectly credible start of a LYN (although I did resent how long it took to install Jupyter).

Of course, they do that sort of thing all day, so they're quick at it. I can do something distantly similar with Rel for crunching through tabular data in CSV files and spreadsheets, but I wouldn't expect anyone else to do it (except Hugh.)

Likewise for other people and other tools.

But the key ingredient in every case is a lot of practice. If you're expecting some tool to suit your preconceptions so you can be immediately not just productive, but fast, with no prior practice, you may be looking for a chimera unless you create it all (perhaps including the underlying language) yourself.

No, I specifically want language tools that allow expert knowledge to be captured and exposed as the LYN for simpler folk to use. The 3GLs we have are enough (give or take a few tweaks), for the users who put in the 10,000 hours of practice, but the tools we have for creating a new LYN are poor. That's where I think the focus should be.

Again, this completes misses the key distinction between the roles of tool creator and tool user. The 4GL is a tool created for people with business needs who think they're programmers by people who really are programmers but find business stuff boring. You are an oddity: academic, compiler writer, but also application programmer, so perhaps you don't see this. In my experience only 1 in 10 programmers can write tools, and the other 9 really don't want to. The 1 in 10 really want all the stuff you've been banging on about, but the other 9 just want to churn out working code with the least possible effort. You won't solve this problem by designing esoteric languages exclusively for the high fliers: you need ways to let them efficiently build tools for everyone else. That's what a 4GL really is (or might have been): a way to divide tool builders from tool users.

I don't think I'm describing anything esoteric. I was actually thinking more along the lines of a better Python (in concept, not necessarily in syntax), but with greater ease and safety, and friendly access to higher-order capability and abstraction. If we were just going for esoteric, high-flying purity, we'd just pick Haskell or OCaml or whatever and be done with it.

But I'm thinking general-purpose abstraction rather than domain-specific predefinition. Specificity should be a feature of libraries written using the language, not features wired into the language.

Wiring features into languages is where 4GLs went wrong, and where they are still wrong. 4GLs invariably reflect a certain arrogance on the part of the 4GL vendor, that its users will only ever want to do things the way the 4GL wants them done. In reality, that almost invariably flies against what the developer wants to do -- even for developers who have not the slightest interest in anything but solving business problems.

That's why 4GLs are all but gone, 3GLs are pervasive, and everyone and their dog thinks learning Python is worthwhile.

You're getting closer. Python is still a crap language, but it makes great glue. Still way too much accidental complexity, but it's a step in the right direction.

Yes, 4GLs phase 1 are gone, but that was just a warmup. My LYN:

has zero accidental complexity

fully expresses the things that need to be done for business purpose

is maintainable and extensible as needed

maps readily unto the underlying 3GL(s)

achieves shorter, safer, higher.

Isn't that just a 'new' 4GL by another name?

Maybe, but I'm not convinced that "maps readily unto the underlying 3GL" is anywhere near as good as "3GL with zero accidental complexity, that can fully express the things that need to be done for business purpose via syntactically-friendly libraries, is maintainable and extensible as needed, and achieves what the user likes."

"What the user likes" is a (maybe the) key ingredient, and it isn't necessarily shorter, safer (sigh -- I have many debates with proponents of dynamic typing), or higher. It might include (or be) other qualities -- like fun to use -- that are much more persuasive in getting users to switch from whatever they use now.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org

#12 · March 16, 2021, 6:34 pm

Quote from Dave Voorhis on March 16, 2021, 6:34 pm

Quote from tobega on March 16, 2021, 12:27 pm

Quote from dandl on March 16, 2021, 9:59 am

Quote from tobega on March 15, 2021, 4:20 pm

How is html even remotely like being "the assembly language of the web"? It's a declarative description of a web page (and increasingly even desktop and mobile app user interfaces). If you want the dumbest data app possible, you could use the html form as a specification and generate everything else.

I know what it is: the lowest possible level description of a browser page, in a form directly interpreted by the browser and from which the DOM is built. Likewise, CSS is the lowest possible description of choices the browser can make in rendering HTML pages, and JS is the lowest possible level description of logic the browser can execute to manipulate the DOM.

These are like protocols or serialisation formats: designed to be rigorously specified for communicating between programs running in different locations or at different times. By contrast a programming language is one designed for the convenience of programmers, in the expectation of an intermediate step (compilation) before being used by the machine. HTML was never designed to be written by humans, and the sooner we hide it inside something else the better.

Twaddle. You will be hard pressed to express this clearer and more compactly.

Even non-coders tend to enjoy playing a little with html as they can see the results immediately and it is very forgiving. I'm not even sure a GUI editor would be much simpler. Added bonus of having accessibility pretty much built in if you stick to semantic elements and don't try to pixel-adjust things.

Indeed. But hopefully they get over it.

I believe anybody doing any kind of coding would never want to learn yet another language instead of just doing html.

You just met one. I have written many things for the Web, but I haven't written HTML for many years and hopefully never will again. Fragments of JSX if I have to, but this too shall pass.

Edited (misread first): For some strange reason many devs seem to incorrectly prefer javascript over html. Just write the html and don't write so much code. Interestingly you are now advocating a strictly inferior and more low-level approach entrenched in a 3G language, contrary to what you seem to be doing otherwise.

Anyway, I thought the premise was to provide for non-programmers? They actually like html.

Yes. HTML (followed shortly thereafter by CSS) is often where the non-professional-programmer gets his or her technical start, and sometimes where he/she stays and turns pro by becoming a UX/UI designer.

I don't think some alternative UI design language, even with an intentional data-entry forms bias (and again, the whole data-entry-via-forms thing is going away) is going to do anything but make the target market wonder why UIs aren't defined, or at least editable, as HTML/CSS.

(The developers who prefer JavaScript over HTML often seem to be the same ones who favour record-by-agonising-record via multiple nested, stateful loops deep over .NET LINQ or C# streams, and prefer ORMs over SQL but only to retrieve all the records so they can be processed -- as the good lord intended -- record-by-agonising-record.)

Quote from tobega on March 16, 2021, 12:27 pm

Quote from dandl on March 16, 2021, 9:59 am

Quote from tobega on March 15, 2021, 4:20 pm

How is html even remotely like being "the assembly language of the web"? It's a declarative description of a web page (and increasingly even desktop and mobile app user interfaces). If you want the dumbest data app possible, you could use the html form as a specification and generate everything else.

I know what it is: the lowest possible level description of a browser page, in a form directly interpreted by the browser and from which the DOM is built. Likewise, CSS is the lowest possible description of choices the browser can make in rendering HTML pages, and JS is the lowest possible level description of logic the browser can execute to manipulate the DOM.

These are like protocols or serialisation formats: designed to be rigorously specified for communicating between programs running in different locations or at different times. By contrast a programming language is one designed for the convenience of programmers, in the expectation of an intermediate step (compilation) before being used by the machine. HTML was never designed to be written by humans, and the sooner we hide it inside something else the better.

Twaddle. You will be hard pressed to express this clearer and more compactly.

Even non-coders tend to enjoy playing a little with html as they can see the results immediately and it is very forgiving. I'm not even sure a GUI editor would be much simpler. Added bonus of having accessibility pretty much built in if you stick to semantic elements and don't try to pixel-adjust things.

Indeed. But hopefully they get over it.

I believe anybody doing any kind of coding would never want to learn yet another language instead of just doing html.

You just met one. I have written many things for the Web, but I haven't written HTML for many years and hopefully never will again. Fragments of JSX if I have to, but this too shall pass.

Edited (misread first): For some strange reason many devs seem to incorrectly prefer javascript over html. Just write the html and don't write so much code. Interestingly you are now advocating a strictly inferior and more low-level approach entrenched in a 3G language, contrary to what you seem to be doing otherwise.

Anyway, I thought the premise was to provide for non-programmers? They actually like html.

Yes. HTML (followed shortly thereafter by CSS) is often where the non-professional-programmer gets his or her technical start, and sometimes where he/she stays and turns pro by becoming a UX/UI designer.

I don't think some alternative UI design language, even with an intentional data-entry forms bias (and again, the whole data-entry-via-forms thing is going away) is going to do anything but make the target market wonder why UIs aren't defined, or at least editable, as HTML/CSS.

(The developers who prefer JavaScript over HTML often seem to be the same ones who favour record-by-agonising-record via multiple nested, stateful loops deep over .NET LINQ or C# streams, and prefer ORMs over SQL but only to retrieve all the records so they can be processed -- as the good lord intended -- record-by-agonising-record.)

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org

#13 · March 16, 2021, 10:46 pm

Quote from dandl on March 16, 2021, 10:46 pm

Quote from tobega on March 16, 2021, 12:27 pm

Quote from dandl on March 16, 2021, 9:59 am

Quote from tobega on March 15, 2021, 4:20 pm

How is html even remotely like being "the assembly language of the web"? It's a declarative description of a web page (and increasingly even desktop and mobile app user interfaces). If you want the dumbest data app possible, you could use the html form as a specification and generate everything else.

I know what it is: the lowest possible level description of a browser page, in a form directly interpreted by the browser and from which the DOM is built. Likewise, CSS is the lowest possible description of choices the browser can make in rendering HTML pages, and JS is the lowest possible level description of logic the browser can execute to manipulate the DOM.

These are like protocols or serialisation formats: designed to be rigorously specified for communicating between programs running in different locations or at different times. By contrast a programming language is one designed for the convenience of programmers, in the expectation of an intermediate step (compilation) before being used by the machine. HTML was never designed to be written by humans, and the sooner we hide it inside something else the better.

Twaddle. You will be hard pressed to express this clearer and more compactly.

Twaddle. This sentence is meaningless.

Even non-coders tend to enjoy playing a little with html as they can see the results immediately and it is very forgiving. I'm not even sure a GUI editor would be much simpler. Added bonus of having accessibility pretty much built in if you stick to semantic elements and don't try to pixel-adjust things.

Indeed. But hopefully they get over it.

I believe anybody doing any kind of coding would never want to learn yet another language instead of just doing html.

You just met one. I have written many things for the Web, but I haven't written HTML for many years and hopefully never will again. Fragments of JSX if I have to, but this too shall pass.

Edited (misread first): For some strange reason many devs seem to incorrectly prefer javascript over html. Just write the html and don't write so much code. Interestingly you are now advocating a strictly inferior and more low-level approach entrenched in a 3G language, contrary to what you seem to be doing otherwise.

Anyway, I thought the premise was to provide for non-programmers? They actually like html.

The premise is shorter-safer-higher. Nobody wants to write 1000 lines of HTML when 100 lines of something else will do the same job, with less room for errors.

So that's that squared away, now about the post-SQL, post-D way of querying, aggregating and analyzing data:

Obviously it will be coupled to some html templating mechanism to present the output, with a sensible default if no template is given.

No, it won't.

The results should be possible to process by general programming constructs, as needed, both during and after the query.

This is too vague to comment on.

Let's also say that simple queries should be dead simple and declarative.

Of course: shorter, safer, higher.

But what would the language look like? Can we integrate something Datalog-like to produce streams of values without turning the whole language into Datalog?

Best not. Datalog is chock full of accidental complexity, and desperately in need of the same approach I outline elsewhere.

Query by example is perhaps more what I was aiming for. Pretty much the same as "calling" a datalog rule. (Addendum: Come to think of it, defining a Datalog rule is pretty much like defining your "while" thing. Datalog is not Turing complete, so it is more of a "safe" language as you were aspiring to. Still might be designed differently, perhaps)

Go ahead, build your approach, then we may be struck by your genius. So far I can't follow.

Datalog with negation has recursion built in, and is at exactly the same computational level as my 9 operator Extended RA. It can be used as a DSL (for the data-centric parts of an app), but is not well suited to writing other parts. And it has no meta-programming as such.

Quote from tobega on March 16, 2021, 12:27 pm

Quote from dandl on March 16, 2021, 9:59 am

Quote from tobega on March 15, 2021, 4:20 pm

How is html even remotely like being "the assembly language of the web"? It's a declarative description of a web page (and increasingly even desktop and mobile app user interfaces). If you want the dumbest data app possible, you could use the html form as a specification and generate everything else.

I know what it is: the lowest possible level description of a browser page, in a form directly interpreted by the browser and from which the DOM is built. Likewise, CSS is the lowest possible description of choices the browser can make in rendering HTML pages, and JS is the lowest possible level description of logic the browser can execute to manipulate the DOM.

These are like protocols or serialisation formats: designed to be rigorously specified for communicating between programs running in different locations or at different times. By contrast a programming language is one designed for the convenience of programmers, in the expectation of an intermediate step (compilation) before being used by the machine. HTML was never designed to be written by humans, and the sooner we hide it inside something else the better.

Twaddle. You will be hard pressed to express this clearer and more compactly.

Twaddle. This sentence is meaningless.

Even non-coders tend to enjoy playing a little with html as they can see the results immediately and it is very forgiving. I'm not even sure a GUI editor would be much simpler. Added bonus of having accessibility pretty much built in if you stick to semantic elements and don't try to pixel-adjust things.

Indeed. But hopefully they get over it.

I believe anybody doing any kind of coding would never want to learn yet another language instead of just doing html.

You just met one. I have written many things for the Web, but I haven't written HTML for many years and hopefully never will again. Fragments of JSX if I have to, but this too shall pass.

Edited (misread first): For some strange reason many devs seem to incorrectly prefer javascript over html. Just write the html and don't write so much code. Interestingly you are now advocating a strictly inferior and more low-level approach entrenched in a 3G language, contrary to what you seem to be doing otherwise.

Anyway, I thought the premise was to provide for non-programmers? They actually like html.

The premise is shorter-safer-higher. Nobody wants to write 1000 lines of HTML when 100 lines of something else will do the same job, with less room for errors.

So that's that squared away, now about the post-SQL, post-D way of querying, aggregating and analyzing data:

Obviously it will be coupled to some html templating mechanism to present the output, with a sensible default if no template is given.

No, it won't.

The results should be possible to process by general programming constructs, as needed, both during and after the query.

This is too vague to comment on.

Let's also say that simple queries should be dead simple and declarative.

Of course: shorter, safer, higher.

But what would the language look like? Can we integrate something Datalog-like to produce streams of values without turning the whole language into Datalog?

Best not. Datalog is chock full of accidental complexity, and desperately in need of the same approach I outline elsewhere.

Query by example is perhaps more what I was aiming for. Pretty much the same as "calling" a datalog rule. (Addendum: Come to think of it, defining a Datalog rule is pretty much like defining your "while" thing. Datalog is not Turing complete, so it is more of a "safe" language as you were aspiring to. Still might be designed differently, perhaps)

Go ahead, build your approach, then we may be struck by your genius. So far I can't follow.

Datalog with negation has recursion built in, and is at exactly the same computational level as my 9 operator Extended RA. It can be used as a DSL (for the data-centric parts of an app), but is not well suited to writing other parts. And it has no meta-programming as such.

Andl - A New Database Language - andl.org

#14 · March 16, 2021, 11:07 pm

Quote from Dave Voorhis on March 16, 2021, 11:07 pm

Quote from dandl on March 16, 2021, 10:46 pm

Quote from tobega on March 16, 2021, 12:27 pm

Quote from dandl on March 16, 2021, 9:59 am

Quote from tobega on March 15, 2021, 4:20 pm

How is html even remotely like being "the assembly language of the web"? It's a declarative description of a web page (and increasingly even desktop and mobile app user interfaces). If you want the dumbest data app possible, you could use the html form as a specification and generate everything else.

I know what it is: the lowest possible level description of a browser page, in a form directly interpreted by the browser and from which the DOM is built. Likewise, CSS is the lowest possible description of choices the browser can make in rendering HTML pages, and JS is the lowest possible level description of logic the browser can execute to manipulate the DOM.

These are like protocols or serialisation formats: designed to be rigorously specified for communicating between programs running in different locations or at different times. By contrast a programming language is one designed for the convenience of programmers, in the expectation of an intermediate step (compilation) before being used by the machine. HTML was never designed to be written by humans, and the sooner we hide it inside something else the better.

Twaddle. You will be hard pressed to express this clearer and more compactly.

Twaddle. This sentence is meaningless.

Even non-coders tend to enjoy playing a little with html as they can see the results immediately and it is very forgiving. I'm not even sure a GUI editor would be much simpler. Added bonus of having accessibility pretty much built in if you stick to semantic elements and don't try to pixel-adjust things.

Indeed. But hopefully they get over it.

I believe anybody doing any kind of coding would never want to learn yet another language instead of just doing html.

You just met one. I have written many things for the Web, but I haven't written HTML for many years and hopefully never will again. Fragments of JSX if I have to, but this too shall pass.

Edited (misread first): For some strange reason many devs seem to incorrectly prefer javascript over html. Just write the html and don't write so much code. Interestingly you are now advocating a strictly inferior and more low-level approach entrenched in a 3G language, contrary to what you seem to be doing otherwise.

Anyway, I thought the premise was to provide for non-programmers? They actually like html.

The premise is shorter-safer-higher. Nobody wants to write 1000 lines of HTML when 100 lines of something else will do the same job, with less room for errors.

Those who do HTML-first (if I may call it that) do it with 0 lines of code using Balsamiq or Axure or equivalent, or they prefer to write 1000 lines of HTML/CSS. Either way, it's to achieve an intended look-and-feel.

If it's acceptable for the look-and-feel to be constrained to some subset of HTML/CSS capability, then that's fine -- until you need some portion of HTML/CSS capability that your tool didn't provide.

The problem here is that the pervasiveness of HTML/CSS simultaneously defines broad expectations in control over style, and narrowly enforces an annoying lack of capability (e.g., specify a set of three cascading or otherwise interdependent drop-down comboboxes, where -- in addition -- the selections in two of the comboboxes are partially filtered by the contents of a textbox if a checkbox is ticked, etc.) that can only be reasonably overcome with browser-side JavaScript (which may have to communicate dynamically with the server-side.)

It's not good, but it's almost unavoidable.

Quote from dandl on March 16, 2021, 10:46 pm

Quote from tobega on March 16, 2021, 12:27 pm

Quote from dandl on March 16, 2021, 9:59 am

Quote from tobega on March 15, 2021, 4:20 pm

How is html even remotely like being "the assembly language of the web"? It's a declarative description of a web page (and increasingly even desktop and mobile app user interfaces). If you want the dumbest data app possible, you could use the html form as a specification and generate everything else.

I know what it is: the lowest possible level description of a browser page, in a form directly interpreted by the browser and from which the DOM is built. Likewise, CSS is the lowest possible description of choices the browser can make in rendering HTML pages, and JS is the lowest possible level description of logic the browser can execute to manipulate the DOM.

These are like protocols or serialisation formats: designed to be rigorously specified for communicating between programs running in different locations or at different times. By contrast a programming language is one designed for the convenience of programmers, in the expectation of an intermediate step (compilation) before being used by the machine. HTML was never designed to be written by humans, and the sooner we hide it inside something else the better.

Twaddle. You will be hard pressed to express this clearer and more compactly.

Twaddle. This sentence is meaningless.

Even non-coders tend to enjoy playing a little with html as they can see the results immediately and it is very forgiving. I'm not even sure a GUI editor would be much simpler. Added bonus of having accessibility pretty much built in if you stick to semantic elements and don't try to pixel-adjust things.

Indeed. But hopefully they get over it.

I believe anybody doing any kind of coding would never want to learn yet another language instead of just doing html.

You just met one. I have written many things for the Web, but I haven't written HTML for many years and hopefully never will again. Fragments of JSX if I have to, but this too shall pass.

Edited (misread first): For some strange reason many devs seem to incorrectly prefer javascript over html. Just write the html and don't write so much code. Interestingly you are now advocating a strictly inferior and more low-level approach entrenched in a 3G language, contrary to what you seem to be doing otherwise.

Anyway, I thought the premise was to provide for non-programmers? They actually like html.

The premise is shorter-safer-higher. Nobody wants to write 1000 lines of HTML when 100 lines of something else will do the same job, with less room for errors.

Those who do HTML-first (if I may call it that) do it with 0 lines of code using Balsamiq or Axure or equivalent, or they prefer to write 1000 lines of HTML/CSS. Either way, it's to achieve an intended look-and-feel.

If it's acceptable for the look-and-feel to be constrained to some subset of HTML/CSS capability, then that's fine -- until you need some portion of HTML/CSS capability that your tool didn't provide.

The problem here is that the pervasiveness of HTML/CSS simultaneously defines broad expectations in control over style, and narrowly enforces an annoying lack of capability (e.g., specify a set of three cascading or otherwise interdependent drop-down comboboxes, where -- in addition -- the selections in two of the comboboxes are partially filtered by the contents of a textbox if a checkbox is ticked, etc.) that can only be reasonably overcome with browser-side JavaScript (which may have to communicate dynamically with the server-side.)

It's not good, but it's almost unavoidable.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org

#15 · March 16, 2021, 11:37 pm

Quote from dandl on March 16, 2021, 11:37 pm

Quote from Dave Voorhis on March 16, 2021, 6:22 pm

Quote from dandl on March 16, 2021, 12:20 am

APL is highest of all, so I'm not sure that metric means much.

I subscribe to the principle: that every major programming project generates its own language, usually a bad one. Otherwise known as LYHINLYN: the language you have is not the language you need (to solve this problem). So first design the TLYN, implement that using TLYH, then solve the problem in your new language. I read that somewhere, but I have no idea where and I can't find it.

Lisp subscribes heavily to that principle. Alan Kay said: "Lisp is a building material, not a programming language".

Notably, Lisp is one of several language families that supports -- and some encourage -- using to create domain-specific languages. Whether syntactically-distinct DSLs are bad, good or indifferent is a matter of some debate (and perhaps personal taste), but it's worth noting that virtually every library of routines or class library is effectively a DSL. They share the syntax of the host language, obviously, but semantically that's what a library is.

Therefore, if you feel the urge to create a syntactically-distinct DSL, the question that comes to mind is what is it about the host language that you don't like?

I'd prefer a seamless, integrated language that supports all the abstraction, expressiveness, and shortcuts that a professional programmer might want, but at the same time supports simple, intuitive, easy syntax for non-professional-programmer developers, so that DSL capability is provided not by defining new syntactically-distinct languages, but by defining libraries that anyone can use.

I don't think these wishes -- for abstract power that professional programmers will like, along with ease of use for beginners and non-professionals -- are contradictory or unreasonable.

Wise words, but you still fail to capture that distinction between tool maker and tool user. Yes, every major app builds up a library of routines in whatever form is available (modules, functions, classes, whatever), and many apps can be written by stringing together calls to those routines in the host language. Please note: the developer and maintainer of those routines takes on the role of tool maker; tool users just use what they're given (and complain when it doesn't work right). This is key.

Python seems part way along that path. I always thought its object orientation was a stumbling block to achieving it, because its OO support is ungainly, half-baked and awkward. It turns out that most non-professional Python programmers never define their own classes, so that's an issue that doesn't come up. But for the beginner/non-professional who becomes proficient enough to want to use classes, they are there.

I imagine the new language we're talking about being the same, but ideally without the flaws: There can be features that the beginner/non-professional might never use, but they are there in case they're wanted. The language should thus support a continuous spectrum of developers -- from rank beginner non-programmers at one end to fully-capable professionals at the other -- with none of the discontinuous breaks in assumed developer ability (or target language) that are inevitably be created by defining a 4GL with "escapes" to (a) 3GL(s).

For context, here is 'hello world':

Java:
class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, World!"); 
    }
}


C:
#include <stdio.h>
int main() {
   printf("Hello World!");
   return 0;
}

Powerflex:

WRITELN "Hello World!"
In each case we have a 5 to one reduction in code lines, at least that in places for bugs and a focus on what is to be done, not comiler cruft. Shorter-safer-higher.

This is my core theme. For this purpose let us assume an underlying Java host language and JVM (the features of C# add nothing to this argument). The question is how to add meta-programming on top of Java as templates were added on top of C++, so that 'hello world' becomes a one liner. [As an aside, the macros in C can do a lot of this and are sorely missed. They are also a major source of hard bugs, so we're not going there.]

So the aims of Java-MP are:

Construction of a DSL from within the language (not by implementing a new compiler)

Code reduction of 5:1 or better, zero accidental complexity

Discoverable, so it can be used by a naive programer with minimal training or documentation

Fully type safe, with full discovery on compile-time errors

Multiple syntax options, not constrained by the host language (syntactical templates)

Open-closed principle: open for extension, closed from modification

Debuggable in the DSL, optional access to underlying Java code

Doesn't sound too hard, does it?

What I'm on about is separating tool building from tool use: making it way easier to create, maintain and extend the LYN. APL is a super example of the LYN for writing certain kinds of mathematical expressions, hampered by some awkward choices of symbols. We should be able to create a 'New APL' easily to replace it, but we can't.

This is mostly an aside, but there is a 'New APL'. It's called J, designed and implemented in part by the same Ken Iverson who invented APL.

From jsoftware.com: "J is a high-level, general-purpose programming language that is particularly suited to the mathematical, statistical, and logical analysis of data. [...] Jd ( J database) is a high-performance columnar RDBMS written in J that is geared toward storing and analyzing large amounts of data. Jd is free for non-commercial use. Jd lives openly and dynamically in the J execution and development environment, so that the full power of J is available to the application developer. For example, Jd columns are mapped to J nouns, so built-in J primitives can apply directly to the data."

Sounds like it might be just what you're looking for.

Yes, I know about J. It's on my list of languages I need to try. But it's not on the meta-programming path.

Ditto for 4GLs, like dBase and Powerflex.

I don't know about Powerflex or Dataflex (which I understand it resembles?) but in the 1980's I created some utilities (in assembly language -- they were Terminate-and-Stay-Resident utilities, back when those were a thing) for a consultant who used Dataflex.

I think that highlights one of the limitations of such languages, which is that its developers inevitably want functionality that they'd like to have been able to create within the language, but couldn't.

(Well, maybe not TSR utilities, as those were a special category of hack, but the principle is good -- the language should be capable, within reason, of doing whatever a developer might want even if the average developer can't.)

Yes, Powerflex is historically compatible with Dataflex, but diverged from it long ago. A skilled programmer can write just about any application program, including Window API access and forms, Web client and server, Unix portable, SQL hosted and queries, COM objects, mountains of stuff. And very substantial meta-programming, via type-sensitive macros with compile-time arithmetic. I had a lot of fun building it, but I don't like programming in it, for all the usual reasons.

Isn't that just a 'new' 4GL by another name?

Maybe, but I'm not convinced that "maps readily unto the underlying 3GL" is anywhere near as good as "3GL with zero accidental complexity, that can fully express the things that need to be done for business purpose via syntactically-friendly libraries, is maintainable and extensible as needed, and achieves what the user likes."

I don't see the distinction, but you did give yourself a challenge: how will you fulfil these aims other than by the kind of language MP extension outlined above?

"What the user likes" is a (maybe the) key ingredient, and it isn't necessarily shorter, safer (sigh -- I have many debates with proponents of dynamic typing), or higher. It might include (or be) other qualities -- like fun to use -- that are much more persuasive in getting users to switch from whatever they use now.

The user likes to get the job done, without pain or undue effort. Other than resisting change, I don't see anyone arguing against shorter-safer-higher. The question is how.

Quote from Dave Voorhis on March 16, 2021, 6:22 pm

Quote from dandl on March 16, 2021, 12:20 am

APL is highest of all, so I'm not sure that metric means much.

I subscribe to the principle: that every major programming project generates its own language, usually a bad one. Otherwise known as LYHINLYN: the language you have is not the language you need (to solve this problem). So first design the TLYN, implement that using TLYH, then solve the problem in your new language. I read that somewhere, but I have no idea where and I can't find it.

Lisp subscribes heavily to that principle. Alan Kay said: "Lisp is a building material, not a programming language".

Notably, Lisp is one of several language families that supports -- and some encourage -- using to create domain-specific languages. Whether syntactically-distinct DSLs are bad, good or indifferent is a matter of some debate (and perhaps personal taste), but it's worth noting that virtually every library of routines or class library is effectively a DSL. They share the syntax of the host language, obviously, but semantically that's what a library is.

Therefore, if you feel the urge to create a syntactically-distinct DSL, the question that comes to mind is what is it about the host language that you don't like?

I'd prefer a seamless, integrated language that supports all the abstraction, expressiveness, and shortcuts that a professional programmer might want, but at the same time supports simple, intuitive, easy syntax for non-professional-programmer developers, so that DSL capability is provided not by defining new syntactically-distinct languages, but by defining libraries that anyone can use.

I don't think these wishes -- for abstract power that professional programmers will like, along with ease of use for beginners and non-professionals -- are contradictory or unreasonable.

Wise words, but you still fail to capture that distinction between tool maker and tool user. Yes, every major app builds up a library of routines in whatever form is available (modules, functions, classes, whatever), and many apps can be written by stringing together calls to those routines in the host language. Please note: the developer and maintainer of those routines takes on the role of tool maker; tool users just use what they're given (and complain when it doesn't work right). This is key.

Python seems part way along that path. I always thought its object orientation was a stumbling block to achieving it, because its OO support is ungainly, half-baked and awkward. It turns out that most non-professional Python programmers never define their own classes, so that's an issue that doesn't come up. But for the beginner/non-professional who becomes proficient enough to want to use classes, they are there.

I imagine the new language we're talking about being the same, but ideally without the flaws: There can be features that the beginner/non-professional might never use, but they are there in case they're wanted. The language should thus support a continuous spectrum of developers -- from rank beginner non-programmers at one end to fully-capable professionals at the other -- with none of the discontinuous breaks in assumed developer ability (or target language) that are inevitably be created by defining a 4GL with "escapes" to (a) 3GL(s).

For context, here is 'hello world':

Java:

class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, World!"); 
    }
}


C:

#include <stdio.h>
int main() {
   printf("Hello World!");
   return 0;
}

Powerflex:

WRITELN "Hello World!"

In each case we have a 5 to one reduction in code lines, at least that in places for bugs and a focus on what is to be done, not comiler cruft. Shorter-safer-higher.

This is my core theme. For this purpose let us assume an underlying Java host language and JVM (the features of C# add nothing to this argument). The question is how to add meta-programming on top of Java as templates were added on top of C++, so that 'hello world' becomes a one liner. [As an aside, the macros in C can do a lot of this and are sorely missed. They are also a major source of hard bugs, so we're not going there.]

So the aims of Java-MP are:

Construction of a DSL from within the language (not by implementing a new compiler)
Code reduction of 5:1 or better, zero accidental complexity
Discoverable, so it can be used by a naive programer with minimal training or documentation
Fully type safe, with full discovery on compile-time errors
Multiple syntax options, not constrained by the host language (syntactical templates)
Open-closed principle: open for extension, closed from modification
Debuggable in the DSL, optional access to underlying Java code

Doesn't sound too hard, does it?

What I'm on about is separating tool building from tool use: making it way easier to create, maintain and extend the LYN. APL is a super example of the LYN for writing certain kinds of mathematical expressions, hampered by some awkward choices of symbols. We should be able to create a 'New APL' easily to replace it, but we can't.

This is mostly an aside, but there is a 'New APL'. It's called J, designed and implemented in part by the same Ken Iverson who invented APL.

From jsoftware.com: "J is a high-level, general-purpose programming language that is particularly suited to the mathematical, statistical, and logical analysis of data. [...] Jd ( J database) is a high-performance columnar RDBMS written in J that is geared toward storing and analyzing large amounts of data. Jd is free for non-commercial use. Jd lives openly and dynamically in the J execution and development environment, so that the full power of J is available to the application developer. For example, Jd columns are mapped to J nouns, so built-in J primitives can apply directly to the data."

Sounds like it might be just what you're looking for.

Yes, I know about J. It's on my list of languages I need to try. But it's not on the meta-programming path.

Ditto for 4GLs, like dBase and Powerflex.

I don't know about Powerflex or Dataflex (which I understand it resembles?) but in the 1980's I created some utilities (in assembly language -- they were Terminate-and-Stay-Resident utilities, back when those were a thing) for a consultant who used Dataflex.

I think that highlights one of the limitations of such languages, which is that its developers inevitably want functionality that they'd like to have been able to create within the language, but couldn't.

(Well, maybe not TSR utilities, as those were a special category of hack, but the principle is good -- the language should be capable, within reason, of doing whatever a developer might want even if the average developer can't.)

Yes, Powerflex is historically compatible with Dataflex, but diverged from it long ago. A skilled programmer can write just about any application program, including Window API access and forms, Web client and server, Unix portable, SQL hosted and queries, COM objects, mountains of stuff. And very substantial meta-programming, via type-sensitive macros with compile-time arithmetic. I had a lot of fun building it, but I don't like programming in it, for all the usual reasons.

Isn't that just a 'new' 4GL by another name?

Maybe, but I'm not convinced that "maps readily unto the underlying 3GL" is anywhere near as good as "3GL with zero accidental complexity, that can fully express the things that need to be done for business purpose via syntactically-friendly libraries, is maintainable and extensible as needed, and achieves what the user likes."

I don't see the distinction, but you did give yourself a challenge: how will you fulfil these aims other than by the kind of language MP extension outlined above?

"What the user likes" is a (maybe the) key ingredient, and it isn't necessarily shorter, safer (sigh -- I have many debates with proponents of dynamic typing), or higher. It might include (or be) other qualities -- like fun to use -- that are much more persuasive in getting users to switch from whatever they use now.

The user likes to get the job done, without pain or undue effort. Other than resisting change, I don't see anyone arguing against shorter-safer-higher. The question is how.

Andl - A New Database Language - andl.org

#16 · March 17, 2021, 1:32 pm

The premise is shorter-safer-higher. Nobody wants to write 1000 lines of HTML when 100 lines of something else will do the same job, with less room for errors.

Those who do HTML-first (if I may call it that) do it with 0 lines of code using Balsamiq or Axure or equivalent, or they prefer to write 1000 lines of HTML/CSS. Either way, it's to achieve an intended look-and-feel.

Of course. I'm only railing against authored HTML. HTML as a file interchange format is just fine. Just don't make me read it or write it.

If it's acceptable for the look-and-feel to be constrained to some subset of HTML/CSS capability, then that's fine -- until you need some portion of HTML/CSS capability that your tool didn't provide.

The problem here is that the pervasiveness of HTML/CSS simultaneously defines broad expectations in control over style, and narrowly enforces an annoying lack of capability (e.g., specify a set of three cascading or otherwise interdependent drop-down comboboxes, where -- in addition -- the selections in two of the comboboxes are partially filtered by the contents of a textbox if a checkbox is ticked, etc.) that can only be reasonably overcome with browser-side JavaScript (which may have to communicate dynamically with the server-side.)

It's not good, but it's almost unavoidable.

I don't know anyone doing anything but JS these days. Why do you think NPM is so big, and JS is near the top of the language list?

And that's another bad language in serious need of a makeover (just one, not dozens of them). But that's another story, not my bag.

Andl - A New Database Language - andl.org

#17 · March 17, 2021, 2:57 pm

Quote from Dave Voorhis on March 17, 2021, 2:57 pm

Quote from dandl on March 17, 2021, 1:32 pm

The premise is shorter-safer-higher. Nobody wants to write 1000 lines of HTML when 100 lines of something else will do the same job, with less room for errors.

Those who do HTML-first (if I may call it that) do it with 0 lines of code using Balsamiq or Axure or equivalent, or they prefer to write 1000 lines of HTML/CSS. Either way, it's to achieve an intended look-and-feel.

Of course. I'm only railing against authored HTML. HTML as a file interchange format is just fine. Just don't make me read it or write it.

Really?

These days, there's two ways I do a Web interface: Either bang it out in raw HTML/CSS (+JS if needed), or I don't touch HTML/CSS/JS at all and use my Spoing framework or raw RAP/RWT in Java.

Either are far superior to either yet another inadequate interface definition language, or (ugh) some dire interface painter (though I don't deprecate those who use them and love them), because I can quickly create simple interfaces or implement monstrously complex interaction. The Java solution is far easier for complex interactivity, though.

If it's acceptable for the look-and-feel to be constrained to some subset of HTML/CSS capability, then that's fine -- until you need some portion of HTML/CSS capability that your tool didn't provide.

The problem here is that the pervasiveness of HTML/CSS simultaneously defines broad expectations in control over style, and narrowly enforces an annoying lack of capability (e.g., specify a set of three cascading or otherwise interdependent drop-down comboboxes, where -- in addition -- the selections in two of the comboboxes are partially filtered by the contents of a textbox if a checkbox is ticked, etc.) that can only be reasonably overcome with browser-side JavaScript (which may have to communicate dynamically with the server-side.)

It's not good, but it's almost unavoidable.

I don't know anyone doing anything but JS these days. Why do you think NPM is so big, and JS is near the top of the language list?

And that's another bad language in serious need of a makeover (just one, not dozens of them). But that's another story, not my bag.

I don't know anyone doing JS in the core enterprise space, except for some lightweight instrumentation of Docker containers and the like. They usually use Typescript, though, which is a few thousand percent better than raw JS.

There's a huge hobbyist and small Website JS following, though I do sometimes see it -- outside of core instrumentation -- in peripheral, usually throwaway, we-need-it-for-the-two-week-campaign-then-delete-it type Node projects. It's good for that.

Per the TIOBE Index, C, Java, Python, C++ and C# are the top five (in order), with JavaScript ranked seventh after Visual Basic. That's near the top, but look what languages are at the top...

Typescript is #41, which is a bit unfortunate, but I'm sure it will grow.

Quote from dandl on March 17, 2021, 1:32 pm

The premise is shorter-safer-higher. Nobody wants to write 1000 lines of HTML when 100 lines of something else will do the same job, with less room for errors.

Those who do HTML-first (if I may call it that) do it with 0 lines of code using Balsamiq or Axure or equivalent, or they prefer to write 1000 lines of HTML/CSS. Either way, it's to achieve an intended look-and-feel.

Of course. I'm only railing against authored HTML. HTML as a file interchange format is just fine. Just don't make me read it or write it.

Really?

These days, there's two ways I do a Web interface: Either bang it out in raw HTML/CSS (+JS if needed), or I don't touch HTML/CSS/JS at all and use my Spoing framework or raw RAP/RWT in Java.

Either are far superior to either yet another inadequate interface definition language, or (ugh) some dire interface painter (though I don't deprecate those who use them and love them), because I can quickly create simple interfaces or implement monstrously complex interaction. The Java solution is far easier for complex interactivity, though.

If it's acceptable for the look-and-feel to be constrained to some subset of HTML/CSS capability, then that's fine -- until you need some portion of HTML/CSS capability that your tool didn't provide.

The problem here is that the pervasiveness of HTML/CSS simultaneously defines broad expectations in control over style, and narrowly enforces an annoying lack of capability (e.g., specify a set of three cascading or otherwise interdependent drop-down comboboxes, where -- in addition -- the selections in two of the comboboxes are partially filtered by the contents of a textbox if a checkbox is ticked, etc.) that can only be reasonably overcome with browser-side JavaScript (which may have to communicate dynamically with the server-side.)

It's not good, but it's almost unavoidable.

I don't know anyone doing anything but JS these days. Why do you think NPM is so big, and JS is near the top of the language list?

And that's another bad language in serious need of a makeover (just one, not dozens of them). But that's another story, not my bag.

I don't know anyone doing JS in the core enterprise space, except for some lightweight instrumentation of Docker containers and the like. They usually use Typescript, though, which is a few thousand percent better than raw JS.

There's a huge hobbyist and small Website JS following, though I do sometimes see it -- outside of core instrumentation -- in peripheral, usually throwaway, we-need-it-for-the-two-week-campaign-then-delete-it type Node projects. It's good for that.

Per the TIOBE Index, C, Java, Python, C++ and C# are the top five (in order), with JavaScript ranked seventh after Visual Basic. That's near the top, but look what languages are at the top...

Typescript is #41, which is a bit unfortunate, but I'm sure it will grow.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org

#18 · March 17, 2021, 3:52 pm

Quote from Dave Voorhis on March 17, 2021, 3:52 pm

Quote from dandl on March 16, 2021, 11:37 pm

Quote from Dave Voorhis on March 16, 2021, 6:22 pm

Quote from dandl on March 16, 2021, 12:20 am

APL is highest of all, so I'm not sure that metric means much.

I subscribe to the principle: that every major programming project generates its own language, usually a bad one. Otherwise known as LYHINLYN: the language you have is not the language you need (to solve this problem). So first design the TLYN, implement that using TLYH, then solve the problem in your new language. I read that somewhere, but I have no idea where and I can't find it.

Lisp subscribes heavily to that principle. Alan Kay said: "Lisp is a building material, not a programming language".

Notably, Lisp is one of several language families that supports -- and some encourage -- using to create domain-specific languages. Whether syntactically-distinct DSLs are bad, good or indifferent is a matter of some debate (and perhaps personal taste), but it's worth noting that virtually every library of routines or class library is effectively a DSL. They share the syntax of the host language, obviously, but semantically that's what a library is.

Therefore, if you feel the urge to create a syntactically-distinct DSL, the question that comes to mind is what is it about the host language that you don't like?

I'd prefer a seamless, integrated language that supports all the abstraction, expressiveness, and shortcuts that a professional programmer might want, but at the same time supports simple, intuitive, easy syntax for non-professional-programmer developers, so that DSL capability is provided not by defining new syntactically-distinct languages, but by defining libraries that anyone can use.

I don't think these wishes -- for abstract power that professional programmers will like, along with ease of use for beginners and non-professionals -- are contradictory or unreasonable.

Wise words, but you still fail to capture that distinction between tool maker and tool user.

I don't consider it to be a significant distinction, except at rather narrow extremes -- with the laboriously-write-a-simple-script non-programmer at one end and the full-time compiler developer at the other. In between is everybody else, where all programmers are a mix of tool developers and tool users.

Indeed, a friend of mine who would be the last to describe himself as a programmer delights in telling tales of an early job he had writing SAS code to write SAS code, because (as a non-programmer) he found writing SAS code to solve problems too dull and tedious, so he wrote SAS programs to generate SAS programs to solve problems.

Was he a tool user?

Or a tool developer?

I don't know, and I'm not sure it matters. A good language should support both. An excellent language should encourage both using just the language.
Yes, every major app builds up a library of routines in whatever form is available (modules, functions, classes, whatever), and many apps can be written by stringing together calls to those routines in the host language. Please note: the developer and maintainer of those routines takes on the role of tool maker; tool users just use what they're given (and complain when it doesn't work right). This is key.

Python seems part way along that path. I always thought its object orientation was a stumbling block to achieving it, because its OO support is ungainly, half-baked and awkward. It turns out that most non-professional Python programmers never define their own classes, so that's an issue that doesn't come up. But for the beginner/non-professional who becomes proficient enough to want to use classes, they are there.

I imagine the new language we're talking about being the same, but ideally without the flaws: There can be features that the beginner/non-professional might never use, but they are there in case they're wanted. The language should thus support a continuous spectrum of developers -- from rank beginner non-programmers at one end to fully-capable professionals at the other -- with none of the discontinuous breaks in assumed developer ability (or target language) that are inevitably be created by defining a 4GL with "escapes" to (a) 3GL(s).

For context, here is 'hello world':

Java:
class HelloWorld{
    public static void main(String[] args){
        System.out.println("Hello, World!"); 
    }
} C:
#include <stdio.h>
int main() {
   printf("Hello World!");
   return 0;
}

Powerflex:

WRITELN "Hello World!"
In each case we have a 5 to one reduction in code lines, at least that in places for bugs and a focus on what is to be done, not comiler cruft. Shorter-safer-higher.
That's an example usually used to illustrate how C and Java (or C#, just change some capitalisation) are verbose (and therefore wrong) whilst Python or Ruby or JavaScript or <insert language here> are so right.

Of course, they don't mention that:

The Java/C#-ish example embodies the Java/C# philosophy of everything-in-a-class, so main (or Main) is defined in a class, and any or multiple classes can be program entry points in a given project; or

The C example is helpfully explicit about the header file that defines 'printf' (and it's not strictly required in this case), and it clearly shows that main() is a function that returns a value, which is very useful when integrating C programs with shell/Python/whatever scripts to invoke them; or

The (usually Python or Ruby or JavaScript) Powerflex code doesn't indicate the entry point at all, so fundamental semantic information is missing; or

A typical developer might never specify a main (or Main) entry point in their entire careers, either because they use a framework that defines it internally or they never start a project from scratch.

That said, I appreciate the value of clarity and I'm not sure the C or Java/C#-ish examples are particularly clear, and there are no doubt better ways of specifying "this is the (or a) entry point."

But per that consideration, the Powerflex example isn't particularly clear, either. It doesn't show where the entry point is at all, so is it specified elsewhere?

Or is every source file an implicit at-the-top entry point (and thus which script acts as the entry point is specified elsewhere?)?

Merely eliding semantics, only to place them elsewhere, is a false "shorter."

Thus, I prefer clarity to merely being "shorter."

This is my core theme. For this purpose let us assume an underlying Java host language and JVM (the features of C# add nothing to this argument). The question is how to add meta-programming on top of Java as templates were added on top of C++, so that 'hello world' becomes a one liner. [As an aside, the macros in C can do a lot of this and are sorely missed. They are also a major source of hard bugs, so we're not going there.]

So the aims of Java-MP are:

Construction of a DSL from within the language (not by implementing a new compiler)

Code reduction of 5:1 or better, zero accidental complexity

Discoverable, so it can be used by a naive programer with minimal training or documentation

Fully type safe, with full discovery on compile-time errors

Multiple syntax options, not constrained by the host language (syntactical templates)

Open-closed principle: open for extension, closed from modification

Debuggable in the DSL, optional access to underlying Java code

Doesn't sound too hard, does it?

That might be a transpiler to Java, assuming you don't define a transpiler as a compiler.

Transpiling languages with almost identical semantics is easy; it's mostly syntactic conversion. Transpiling languages with different semantics can be very hard. I've written a few transpilers, mostly for pedagogy, and it's surprisingly challenging.

I suspect directly compiling to the JVM -- generating bytecode -- or running as an interpreter might be easier (though it is a new compiler.)

Aside from "multiple syntax options", you've essentially described Kotlin. Or Groovy. Or any of the other JVM languages.

I like "Discoverable," though I've found over the years that whilst skilled programmers can easily distinguish intuitive from non-intuitive things for programmers, for non-programmers doing programming there is no such thing as intuitive. It's all equally baffling.

What I'm on about is separating tool building from tool use: making it way easier to create, maintain and extend the LYN. APL is a super example of the LYN for writing certain kinds of mathematical expressions, hampered by some awkward choices of symbols. We should be able to create a 'New APL' easily to replace it, but we can't.

This is mostly an aside, but there is a 'New APL'. It's called J, designed and implemented in part by the same Ken Iverson who invented APL.

From jsoftware.com: "J is a high-level, general-purpose programming language that is particularly suited to the mathematical, statistical, and logical analysis of data. [...] Jd ( J database) is a high-performance columnar RDBMS written in J that is geared toward storing and analyzing large amounts of data. Jd is free for non-commercial use. Jd lives openly and dynamically in the J execution and development environment, so that the full power of J is available to the application developer. For example, Jd columns are mapped to J nouns, so built-in J primitives can apply directly to the data."

Sounds like it might be just what you're looking for.

Yes, I know about J. It's on my list of languages I need to try. But it's not on the meta-programming path.

Ditto for 4GLs, like dBase and Powerflex.

I don't know about Powerflex or Dataflex (which I understand it resembles?) but in the 1980's I created some utilities (in assembly language -- they were Terminate-and-Stay-Resident utilities, back when those were a thing) for a consultant who used Dataflex.

I think that highlights one of the limitations of such languages, which is that its developers inevitably want functionality that they'd like to have been able to create within the language, but couldn't.

(Well, maybe not TSR utilities, as those were a special category of hack, but the principle is good -- the language should be capable, within reason, of doing whatever a developer might want even if the average developer can't.)

Yes, Powerflex is historically compatible with Dataflex, but diverged from it long ago. A skilled programmer can write just about any application program, including Window API access and forms, Web client and server, Unix portable, SQL hosted and queries, COM objects, mountains of stuff. And very substantial meta-programming, via type-sensitive macros with compile-time arithmetic. I had a lot of fun building it, but I don't like programming in it, for all the usual reasons.

It sounds like it has the right semantics. Why not change the syntax so it's more appealing?

Isn't that just a 'new' 4GL by another name?

Maybe, but I'm not convinced that "maps readily unto the underlying 3GL" is anywhere near as good as "3GL with zero accidental complexity, that can fully express the things that need to be done for business purpose via syntactically-friendly libraries, is maintainable and extensible as needed, and achieves what the user likes."

I don't see the distinction, but you did give yourself a challenge: how will you fulfil these aims other than by the kind of language MP extension outlined above?

"What the user likes" is a (maybe the) key ingredient, and it isn't necessarily shorter, safer (sigh -- I have many debates with proponents of dynamic typing), or higher. It might include (or be) other qualities -- like fun to use -- that are much more persuasive in getting users to switch from whatever they use now.

The user likes to get the job done, without pain or undue effort. Other than resisting change, I don't see anyone arguing against shorter-safer-higher. The question is how.

The user likes what's fun. Observation of hundreds of business users over the years tells me that whilst pain and undue effort negatively affect tool appeal, merely being painless and effortless is not enough to make a tool appealing.

It needs to be fun.

Something quite notable about Python is the degree to which beginner or otherwise non-professional programmers will spend endless hours attempting to get buggy Python code working, and enjoy it.

There's a similar enjoyment I've observed in Excel users. They may be, at best, marginally productive, but as long as they're having fun, it's good.

Quote from dandl on March 16, 2021, 11:37 pm

Quote from Dave Voorhis on March 16, 2021, 6:22 pm

Quote from dandl on March 16, 2021, 12:20 am

APL is highest of all, so I'm not sure that metric means much.

I subscribe to the principle: that every major programming project generates its own language, usually a bad one. Otherwise known as LYHINLYN: the language you have is not the language you need (to solve this problem). So first design the TLYN, implement that using TLYH, then solve the problem in your new language. I read that somewhere, but I have no idea where and I can't find it.

Lisp subscribes heavily to that principle. Alan Kay said: "Lisp is a building material, not a programming language".

Notably, Lisp is one of several language families that supports -- and some encourage -- using to create domain-specific languages. Whether syntactically-distinct DSLs are bad, good or indifferent is a matter of some debate (and perhaps personal taste), but it's worth noting that virtually every library of routines or class library is effectively a DSL. They share the syntax of the host language, obviously, but semantically that's what a library is.

Therefore, if you feel the urge to create a syntactically-distinct DSL, the question that comes to mind is what is it about the host language that you don't like?

I'd prefer a seamless, integrated language that supports all the abstraction, expressiveness, and shortcuts that a professional programmer might want, but at the same time supports simple, intuitive, easy syntax for non-professional-programmer developers, so that DSL capability is provided not by defining new syntactically-distinct languages, but by defining libraries that anyone can use.

I don't think these wishes -- for abstract power that professional programmers will like, along with ease of use for beginners and non-professionals -- are contradictory or unreasonable.

Wise words, but you still fail to capture that distinction between tool maker and tool user.

I don't consider it to be a significant distinction, except at rather narrow extremes -- with the laboriously-write-a-simple-script non-programmer at one end and the full-time compiler developer at the other. In between is everybody else, where all programmers are a mix of tool developers and tool users.

Indeed, a friend of mine who would be the last to describe himself as a programmer delights in telling tales of an early job he had writing SAS code to write SAS code, because (as a non-programmer) he found writing SAS code to solve problems too dull and tedious, so he wrote SAS programs to generate SAS programs to solve problems.

Was he a tool user?

Or a tool developer?

I don't know, and I'm not sure it matters. A good language should support both. An excellent language should encourage both using just the language.

Yes, every major app builds up a library of routines in whatever form is available (modules, functions, classes, whatever), and many apps can be written by stringing together calls to those routines in the host language. Please note: the developer and maintainer of those routines takes on the role of tool maker; tool users just use what they're given (and complain when it doesn't work right). This is key.

Python seems part way along that path. I always thought its object orientation was a stumbling block to achieving it, because its OO support is ungainly, half-baked and awkward. It turns out that most non-professional Python programmers never define their own classes, so that's an issue that doesn't come up. But for the beginner/non-professional who becomes proficient enough to want to use classes, they are there.

I imagine the new language we're talking about being the same, but ideally without the flaws: There can be features that the beginner/non-professional might never use, but they are there in case they're wanted. The language should thus support a continuous spectrum of developers -- from rank beginner non-programmers at one end to fully-capable professionals at the other -- with none of the discontinuous breaks in assumed developer ability (or target language) that are inevitably be created by defining a 4GL with "escapes" to (a) 3GL(s).

For context, here is 'hello world':

Java:
class HelloWorld{
    public static void main(String[] args){
        System.out.println("Hello, World!"); 
    }
} C:
#include <stdio.h>
int main() {
   printf("Hello World!");
   return 0;
}

Powerflex:

WRITELN "Hello World!"
In each case we have a 5 to one reduction in code lines, at least that in places for bugs and a focus on what is to be done, not comiler cruft. Shorter-safer-higher.

That's an example usually used to illustrate how C and Java (or C#, just change some capitalisation) are verbose (and therefore wrong) whilst Python or Ruby or JavaScript or <insert language here> are so right.

Of course, they don't mention that:

The Java/C#-ish example embodies the Java/C# philosophy of everything-in-a-class, so main (or Main) is defined in a class, and any or multiple classes can be program entry points in a given project; or
The C example is helpfully explicit about the header file that defines 'printf' (and it's not strictly required in this case), and it clearly shows that main() is a function that returns a value, which is very useful when integrating C programs with shell/Python/whatever scripts to invoke them; or
The (usually Python or Ruby or JavaScript) Powerflex code doesn't indicate the entry point at all, so fundamental semantic information is missing; or
A typical developer might never specify a main (or Main) entry point in their entire careers, either because they use a framework that defines it internally or they never start a project from scratch.

That said, I appreciate the value of clarity and I'm not sure the C or Java/C#-ish examples are particularly clear, and there are no doubt better ways of specifying "this is the (or a) entry point."

But per that consideration, the Powerflex example isn't particularly clear, either. It doesn't show where the entry point is at all, so is it specified elsewhere?

Or is every source file an implicit at-the-top entry point (and thus which script acts as the entry point is specified elsewhere?)?

Merely eliding semantics, only to place them elsewhere, is a false "shorter."

Thus, I prefer clarity to merely being "shorter."

This is my core theme. For this purpose let us assume an underlying Java host language and JVM (the features of C# add nothing to this argument). The question is how to add meta-programming on top of Java as templates were added on top of C++, so that 'hello world' becomes a one liner. [As an aside, the macros in C can do a lot of this and are sorely missed. They are also a major source of hard bugs, so we're not going there.]

So the aims of Java-MP are:

Construction of a DSL from within the language (not by implementing a new compiler)

Code reduction of 5:1 or better, zero accidental complexity

Discoverable, so it can be used by a naive programer with minimal training or documentation

Fully type safe, with full discovery on compile-time errors

Multiple syntax options, not constrained by the host language (syntactical templates)

Open-closed principle: open for extension, closed from modification

Debuggable in the DSL, optional access to underlying Java code

Doesn't sound too hard, does it?

That might be a transpiler to Java, assuming you don't define a transpiler as a compiler.

Transpiling languages with almost identical semantics is easy; it's mostly syntactic conversion. Transpiling languages with different semantics can be very hard. I've written a few transpilers, mostly for pedagogy, and it's surprisingly challenging.

I suspect directly compiling to the JVM -- generating bytecode -- or running as an interpreter might be easier (though it is a new compiler.)

Aside from "multiple syntax options", you've essentially described Kotlin. Or Groovy. Or any of the other JVM languages.

I like "Discoverable," though I've found over the years that whilst skilled programmers can easily distinguish intuitive from non-intuitive things for programmers, for non-programmers doing programming there is no such thing as intuitive. It's all equally baffling.

What I'm on about is separating tool building from tool use: making it way easier to create, maintain and extend the LYN. APL is a super example of the LYN for writing certain kinds of mathematical expressions, hampered by some awkward choices of symbols. We should be able to create a 'New APL' easily to replace it, but we can't.

This is mostly an aside, but there is a 'New APL'. It's called J, designed and implemented in part by the same Ken Iverson who invented APL.

From jsoftware.com: "J is a high-level, general-purpose programming language that is particularly suited to the mathematical, statistical, and logical analysis of data. [...] Jd ( J database) is a high-performance columnar RDBMS written in J that is geared toward storing and analyzing large amounts of data. Jd is free for non-commercial use. Jd lives openly and dynamically in the J execution and development environment, so that the full power of J is available to the application developer. For example, Jd columns are mapped to J nouns, so built-in J primitives can apply directly to the data."

Sounds like it might be just what you're looking for.

Yes, I know about J. It's on my list of languages I need to try. But it's not on the meta-programming path.

Ditto for 4GLs, like dBase and Powerflex.

I don't know about Powerflex or Dataflex (which I understand it resembles?) but in the 1980's I created some utilities (in assembly language -- they were Terminate-and-Stay-Resident utilities, back when those were a thing) for a consultant who used Dataflex.

I think that highlights one of the limitations of such languages, which is that its developers inevitably want functionality that they'd like to have been able to create within the language, but couldn't.

(Well, maybe not TSR utilities, as those were a special category of hack, but the principle is good -- the language should be capable, within reason, of doing whatever a developer might want even if the average developer can't.)

Yes, Powerflex is historically compatible with Dataflex, but diverged from it long ago. A skilled programmer can write just about any application program, including Window API access and forms, Web client and server, Unix portable, SQL hosted and queries, COM objects, mountains of stuff. And very substantial meta-programming, via type-sensitive macros with compile-time arithmetic. I had a lot of fun building it, but I don't like programming in it, for all the usual reasons.

It sounds like it has the right semantics. Why not change the syntax so it's more appealing?

Isn't that just a 'new' 4GL by another name?

Maybe, but I'm not convinced that "maps readily unto the underlying 3GL" is anywhere near as good as "3GL with zero accidental complexity, that can fully express the things that need to be done for business purpose via syntactically-friendly libraries, is maintainable and extensible as needed, and achieves what the user likes."

I don't see the distinction, but you did give yourself a challenge: how will you fulfil these aims other than by the kind of language MP extension outlined above?

"What the user likes" is a (maybe the) key ingredient, and it isn't necessarily shorter, safer (sigh -- I have many debates with proponents of dynamic typing), or higher. It might include (or be) other qualities -- like fun to use -- that are much more persuasive in getting users to switch from whatever they use now.

The user likes to get the job done, without pain or undue effort. Other than resisting change, I don't see anyone arguing against shorter-safer-higher. The question is how.

The user likes what's fun. Observation of hundreds of business users over the years tells me that whilst pain and undue effort negatively affect tool appeal, merely being painless and effortless is not enough to make a tool appealing.

It needs to be fun.

Something quite notable about Python is the degree to which beginner or otherwise non-professional programmers will spend endless hours attempting to get buggy Python code working, and enjoy it.

There's a similar enjoyment I've observed in Excel users. They may be, at best, marginally productive, but as long as they're having fun, it's good.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org

#19 · March 17, 2021, 7:14 pm

Quote from Dave Voorhis on March 16, 2021, 11:07 pm

The problem here is that the pervasiveness of HTML/CSS simultaneously defines broad expectations in control over style, and narrowly enforces an annoying lack of capability (e.g., specify a set of three cascading or otherwise interdependent drop-down comboboxes, where -- in addition -- the selections in two of the comboboxes are partially filtered by the contents of a textbox if a checkbox is ticked, etc.) that can only be reasonably overcome with browser-side JavaScript (which may have to communicate dynamically with the server-side.)

It's not good, but it's almost unavoidable.

The "annoying lack of capability" is exclusively and entirely a consequence of the fact that browsers do not create the entire set of GUI facilities for their plugins that windowing desktop OS's do for theirs. And that doing so (which would mean effectively turn the browser into a desktop OS of its own) means creating security problems that even the "native" desktop OS environments do not know how to handle appropriately because none (not a single one) of the so-called "software engineers" who created those desktop environments ever bothered to look at how security was treated on real computers (i.e. the mainframe).

The point being : it ***IS*** avoidable. Just look at how it is done (still to this very day) on real computers. (Yes, it means there ***IS*** "install procedure" to go through for any software you want to use and subsequently there ***IS*** "configuration procedure" to go through before starting to use it, but does the cost of that offset the weighed risk of not knowing who -which unauthenticated parties, that is- has access to any of the data managed by your software ?)

#20 · March 17, 2021, 8:44 pm

Quote from Dave Voorhis on March 17, 2021, 8:44 pm

Quote from Erwin on March 17, 2021, 7:14 pm

Quote from Dave Voorhis on March 16, 2021, 11:07 pm

The problem here is that the pervasiveness of HTML/CSS simultaneously defines broad expectations in control over style, and narrowly enforces an annoying lack of capability (e.g., specify a set of three cascading or otherwise interdependent drop-down comboboxes, where -- in addition -- the selections in two of the comboboxes are partially filtered by the contents of a textbox if a checkbox is ticked, etc.) that can only be reasonably overcome with browser-side JavaScript (which may have to communicate dynamically with the server-side.)

It's not good, but it's almost unavoidable.

The "annoying lack of capability" is exclusively and entirely a consequence of the fact that browsers do not create the entire set of GUI facilities for their plugins that windowing desktop OS's do for theirs. And that doing so (which would mean effectively turn the browser into a desktop OS of its own) means creating security problems that even the "native" desktop OS environments do not know how to handle appropriately because none (not a single one) of the so-called "software engineers" who created those desktop environments ever bothered to look at how security was treated on real computers (i.e. the mainframe).

The point being : it ***IS*** avoidable. Just look at how it is done (still to this very day) on real computers. (Yes, it means there ***IS*** "install procedure" to go through for any software you want to use and subsequently there ***IS*** "configuration procedure" to go through before starting to use it, but does the cost of that offset the weighed risk of not knowing who -which unauthenticated parties, that is- has access to any of the data managed by your software ?)

We're in a world now where typical database management is handed to a 3rd party in "the cloud", the backend server is a "cloud" instance hosted by the same or a different 3rd party, and the application front-end runs in whatever Web browser the user has on his home computer, office computer (pre-COVID), mobile phone, tablet, you-name-it.

Security is defined around the RESTful interface provided by the application(s) on the server. The HTTP-transferable HTML/CSS/JS client-side need not be particularly secured, because it's worthless if you aren't granted access to the RESTful interface and you can't trust the client-side anyway.

The install and configure procedure is just part of the application code deployed via the CI/CD chain, the output of which points only at that server (or a scalable cluster thereof) and its lower test environments.

For better or worse, that's "real computers" in these parts.

Quote from Erwin on March 17, 2021, 7:14 pm

Quote from Dave Voorhis on March 16, 2021, 11:07 pm

The problem here is that the pervasiveness of HTML/CSS simultaneously defines broad expectations in control over style, and narrowly enforces an annoying lack of capability (e.g., specify a set of three cascading or otherwise interdependent drop-down comboboxes, where -- in addition -- the selections in two of the comboboxes are partially filtered by the contents of a textbox if a checkbox is ticked, etc.) that can only be reasonably overcome with browser-side JavaScript (which may have to communicate dynamically with the server-side.)

It's not good, but it's almost unavoidable.

The "annoying lack of capability" is exclusively and entirely a consequence of the fact that browsers do not create the entire set of GUI facilities for their plugins that windowing desktop OS's do for theirs. And that doing so (which would mean effectively turn the browser into a desktop OS of its own) means creating security problems that even the "native" desktop OS environments do not know how to handle appropriately because none (not a single one) of the so-called "software engineers" who created those desktop environments ever bothered to look at how security was treated on real computers (i.e. the mainframe).

The point being : it ***IS*** avoidable. Just look at how it is done (still to this very day) on real computers. (Yes, it means there ***IS*** "install procedure" to go through for any software you want to use and subsequently there ***IS*** "configuration procedure" to go through before starting to use it, but does the cost of that offset the weighed risk of not knowing who -which unauthenticated parties, that is- has access to any of the data managed by your software ?)

We're in a world now where typical database management is handed to a 3rd party in "the cloud", the backend server is a "cloud" instance hosted by the same or a different 3rd party, and the application front-end runs in whatever Web browser the user has on his home computer, office computer (pre-COVID), mobile phone, tablet, you-name-it.

Security is defined around the RESTful interface provided by the application(s) on the server. The HTTP-transferable HTML/CSS/JS client-side need not be particularly secured, because it's worthless if you aren't granted access to the RESTful interface and you can't trust the client-side anyway.

The install and configure procedure is just part of the application code deployed via the CI/CD chain, the output of which points only at that server (or a scalable cluster thereof) and its lower test environments.

For better or worse, that's "real computers" in these parts.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org

The Forum for Discussion about The Third Manifesto and Related Matters

Life after D