The Forum for Discussion about The Third Manifesto and Related Matters

Please or Register to create posts and topics.

Language safety missing when D meets Database

PreviousPage 2 of 3Next
Quote from AntC on June 11, 2021, 10:40 am
Quote from dandl on June 10, 2021, 4:16 am
Quote from Dave Voorhis on June 9, 2021, 2:30 pm

Rel is notionally (or at least started out as) a closed environment akin to Smalltalk, where the code is stored in the database and all dependencies are managed to ensure that if it compiles, it runs.

But then to make it practically useful, I extended it to reference external dependencies like SQL DBMSs via JDBC, CSV files and the like. Utility has obviously increased, but I can no longer guarantee that what compiles, runs. I can guarantee that a native Rel relvar can't be dropped until it has no code references, but I can't guarantee that a JDBC connection to an intermittently-available SQL DBMS which worked yesterday will work today. That currently means a run-time failure. What it should do -- but doesn't do, yet -- is the compiler should obligate the developer provide some code path(s) to safely handle the situation where the external dependency has failed.

It could do better than that. It could say: your program has a dependency on X and is no longer runnable. Rather than wait to get to that part of the program and failing, it could capture dependencies up front. Yes, a programmer might make a positive decision to run and handle the consequences, but the default should be to explicitly manage and track dependencies.

 

... leave all the responsibility for external dependencies to be resolved by having the programmer add extra code. Knowing what code to write can be extremely challenging.

My question is simply: can we reverse that onus? Can we have languages so designed that the compiler tracks external dependencies and ensures they are handled as we choose, without the program ever failing at runtime? What would it take to achieve that?

The System/38 later re-badged as AS/400 was a database machine. The Operating System knew for each object-database which source files its tables were compiled from; and which source/which schema any application was compiled against. When an application tried to open a table, the O.S. intervened and cross-checked schemas for all tables the application was compiled against. Any mis-matched version anywhere just stopped the application running. The application couldn't get half way through a transaction and then discover its API out of kilter with some called routine. (Schemas for APIs also had to be declared as if database tables, under the same version-control.)

I remember it well. We were looking for an entry point into computer systems for hospitals in the early 80s, and I was very impressed by the AS/400 (RPG not so much). [I also remember a couple of guy in a small booth at the back of the 1980 IFIP computer conference exhibition pitching something on VAX that was 'better than System R', and wondering if it would amount to anything. Oracle, they called themselves -- presumptious I thought.]

Then IBM brought in the i-Series, alleged to be 'super-400', with gosh! SQL. And they sneakily retired all that O.S. control -- because SQL, apparently. Also because internet and the need for interconnectivity to any heap of old iron. Sounds like the same story as Rel.

You were hostile to the idea of C macros and quite happy to use reflection. Both are equally unsafe, but at least C macros are transparent and visible: you can see what they claim to do. I would ban reflection, and insist that the relevant information is made available at compile time, but safely.

Neither macros nor reflection are proper version-control integrity; they rely on the programmer thinking to check what is (in their view) significant for safety.

And then reacting with surprise when something falls over, and scrambling to find it and fix it under pressure. A good idea? Not!

It occurred to me that the languages I use to express the logic of my programs have advanced immeasurably. From pointers and words through fixed types to records, declared types, OOF (or FP), templates etc is an uplift of at least 10x, IMO. But the very first program I ever wrote with access to a file system did exactly the same as they do now if the file is not there: barf! Our programs are still just as unsafe now as they were 40-50 years ago, if you ever try to actually run them. One missing dependency and you're dead.

So this is my theme: 'if it compiles it runs' really means 'safe execution' and that IMO means tracking and managing dependencies. Ideas please.

Andl - A New Database Language - andl.org

With the current popular programming languages, we can almost achieve 'if it compiles it runs' but it requires developer effort -- the compiler will only obligate error handling in some places, not everywhere errors can occur -- and external dependencies have to be treated in either of two ways:

  1. Handle them like user input. I.e., assume the external facility is going to be wrong and/or broken most of the time and act accordingly. Of course, that requires careful design, and all constructs for handling external dependencies will be dynamic. E.g., lots of unpleasantly-verbose and error-prone row.getColumnValue("CustomerAddress") and the like, instead of row.CustomerAddress. Or, if row.CustomerAddress is supported by the language as a dynamic construct (sugared to look static) make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)
  2. Handle them like local dependencies. Create/generate/import static constructs, so you get to use row.CustomerAddress and it is genuinely static, but make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)

We could do better to obligate handling of possible failures -- or alternate code paths -- everywhere they could occur, in both local and external dependencies. That's the notion behind Java checked exceptions. That's why Java's standard database-access mechanism -- JDBC -- can throw a checked (i.e., must be explicitly caught somewhere) SQLException for every operation, and that's why higher-level operations based on JDBC (which is almost all of them) typically either can throw a checked SQLException, or throw some other checked exception that wraps a SQLException.

It works very well when done properly. I can write Java programs whose environment, both local and external, can fail and they'll degrade gracefully and helpfully inform -- via logs at least, and sometimes via helpful messages (where appropriate) to a designated user -- what's failed.

Where that approach goes wrong is when lazy developers create empty exception handling blocks or otherwise naively defeat the mechanism, and the program silently keeps running in a broken state, or crashes, or otherwise fails badly instead of helpfully (i.e., noisily) and gracefully degrading. But at least empty exception handling blocks are relatively easy to locate.

Or, it goes wrong when lazy or naive developers create libraries that should throw checked exceptions when external dependencies break (because they will), but instead throw unchecked (i.e., catching them is not required, and thus the program will crash if they've not caught) exceptions, which are notionally intended for things you can't possibly recover from, like out-of-memory or coding errors. Then you get runtime surprises and unhandled crashes, and that can be harder to fix because you often don't know the problem exists until you encounter it in production.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org
Quote from Dave Voorhis on June 12, 2021, 7:23 pm

With the current popular programming languages, we can almost achieve 'if it compiles it runs' but it requires developer effort -- the compiler will only obligate error handling in some places, not everywhere errors can occur -- and external dependencies have to be treated in either of two ways:

  1. Handle them like user input. I.e., assume the external facility is going to be wrong and/or broken most of the time and act accordingly. Of course, that requires careful design, and all constructs for handling external dependencies will be dynamic. E.g., lots of unpleasantly-verbose and error-prone row.getColumnValue("CustomerAddress") and the like, instead of row.CustomerAddress. Or, if row.CustomerAddress is supported by the language as a dynamic construct (sugared to look static) make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)
  2. Handle them like local dependencies. Create/generate/import static constructs, so you get to use row.CustomerAddress and it is genuinely static, but make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)

We could do better to obligate handling of possible failures -- or alternate code paths -- everywhere they could occur, in both local and external dependencies. That's the notion behind Java checked exceptions. That's why Java's standard database-access mechanism -- JDBC -- can throw a checked (i.e., must be explicitly caught somewhere) SQLException for every operation, and that's why higher-level operations based on JDBC (which is almost all of them) typically either can throw a checked SQLException, or throw some other checked exception that wraps a SQLException.

It works very well when done properly. I can write Java programs whose environment, both local and external, can fail and they'll degrade gracefully and helpfully inform -- via logs at least, and sometimes via helpful messages (where appropriate) to a designated user -- what's failed.

I accept this as a reasonable outline of the status quo. But please note: (a) these are strategies that have changed little in 40-odd years and (b) you can (for Java), but I can't (for Java) and you can't (for language X). It takes deep knowledge of the runtime environment (not just the language).

Where that approach goes wrong is when lazy developers create empty exception handling blocks or otherwise naively defeat the mechanism, and the program silently keeps running in a broken state, or crashes, or otherwise fails badly instead of helpfully (i.e., noisily) and gracefully degrading. But at least empty exception handling blocks are relatively easy to locate.

Or, it goes wrong when lazy or naive developers create libraries that should throw checked exceptions when external dependencies break (because they will), but instead throw unchecked (i.e., catching them is not required, and thus the program will crash if they've not caught) exceptions, which are notionally intended for things you can't possibly recover from, like out-of-memory or coding errors. Then you get runtime surprises and unhandled crashes, and that can be harder to fix because you often don't know the problem exists until you encounter it in production.

It goes wrong when a highly skilled and motivated developer tries to solve problems in new territory, without the time or inclination to first acquire all those new skills. It goes wrong because those strategies and underlying dependencies are not directly transferable but rely on skill and runtime code without static or pre-execution checking or testing. [It relies a lot on SO.]

So, how do we design systems to operate safely in the environment you describe within the range of a competent programmer not possessing requisite platform-specific skills? Assume the programmer is not to use any platform specifics such as exception handling. My suggestion:

  • Identify all the kinds of failure handling we might want to use (prevent execution, abort with log message; retry; substitute default data; substitute previous valid data; etc)
  • For each point of failure implement generic handlers for each kind (platform specific knowledge required for this step)
  • For each point of failure require the programmer to choose how it is to be handled (or default; or multiple) (language extension required)
  • Provide a mechanism to report on points of failure prior or during execution (without waiting until it gets hit).

My point: the new languages we need are those that will support safe execution: if it compiles, it runs.

Andl - A New Database Language - andl.org
Quote from dandl on June 13, 2021, 4:23 am
Quote from Dave Voorhis on June 12, 2021, 7:23 pm

With the current popular programming languages, we can almost achieve 'if it compiles it runs' but it requires developer effort -- the compiler will only obligate error handling in some places, not everywhere errors can occur -- and external dependencies have to be treated in either of two ways:

  1. Handle them like user input. I.e., assume the external facility is going to be wrong and/or broken most of the time and act accordingly. Of course, that requires careful design, and all constructs for handling external dependencies will be dynamic. E.g., lots of unpleasantly-verbose and error-prone row.getColumnValue("CustomerAddress") and the like, instead of row.CustomerAddress. Or, if row.CustomerAddress is supported by the language as a dynamic construct (sugared to look static) make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)
  2. Handle them like local dependencies. Create/generate/import static constructs, so you get to use row.CustomerAddress and it is genuinely static, but make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)

We could do better to obligate handling of possible failures -- or alternate code paths -- everywhere they could occur, in both local and external dependencies. That's the notion behind Java checked exceptions. That's why Java's standard database-access mechanism -- JDBC -- can throw a checked (i.e., must be explicitly caught somewhere) SQLException for every operation, and that's why higher-level operations based on JDBC (which is almost all of them) typically either can throw a checked SQLException, or throw some other checked exception that wraps a SQLException.

It works very well when done properly. I can write Java programs whose environment, both local and external, can fail and they'll degrade gracefully and helpfully inform -- via logs at least, and sometimes via helpful messages (where appropriate) to a designated user -- what's failed.

I accept this as a reasonable outline of the status quo. But please note: (a) these are strategies that have changed little in 40-odd years and (b) you can (for Java), but I can't (for Java) and you can't (for language X). It takes deep knowledge of the runtime environment (not just the language).

It's not so much that it takes deep knowledge of the runtime environment -- which, ideally, it shouldn't require -- but the fact that the failure mechanisms are often hidden and only appear as a runtime surprise rather than being explicit at compile-time, and in particular, without (as it should be) obligating the developer to consider them and handle them (even if badly, which may be better than nothing at all.)

Where that approach goes wrong is when lazy developers create empty exception handling blocks or otherwise naively defeat the mechanism, and the program silently keeps running in a broken state, or crashes, or otherwise fails badly instead of helpfully (i.e., noisily) and gracefully degrading. But at least empty exception handling blocks are relatively easy to locate.

Or, it goes wrong when lazy or naive developers create libraries that should throw checked exceptions when external dependencies break (because they will), but instead throw unchecked (i.e., catching them is not required, and thus the program will crash if they've not caught) exceptions, which are notionally intended for things you can't possibly recover from, like out-of-memory or coding errors. Then you get runtime surprises and unhandled crashes, and that can be harder to fix because you often don't know the problem exists until you encounter it in production.

It goes wrong when a highly skilled and motivated developer tries to solve problems in new territory, without the time or inclination to first acquire all those new skills. It goes wrong because those strategies and underlying dependencies are not directly transferable but rely on skill and runtime code without static or pre-execution checking or testing. [It relies a lot on SO.]

What's an "SO"?

So, how do we design systems to operate safely in the environment you describe within the range of a competent programmer not possessing requisite platform-specific skills? Assume the programmer is not to use any platform specifics such as exception handling. My suggestion:

  • Identify all the kinds of failure handling we might want to use (prevent execution, abort with log message; retry; substitute default data; substitute previous valid data; etc)
  • For each point of failure implement generic handlers for each kind (platform specific knowledge required for this step)
  • For each point of failure require the programmer to choose how it is to be handled (or default; or multiple) (language extension required)
  • Provide a mechanism to report on points of failure prior or during execution (without waiting until it gets hit).

My point: the new languages we need are those that will support safe execution: if it compiles, it runs.

Perhaps the conceptually simplest way is to make every function call return a value that is a union of its explicit return type and types representing all otherwise-unhandled failures, and all union types obligate code paths to handle all components of the union (though not necessarily each individually; default, other or similar is an acceptable case.)  In this scheme, there are no alternate code paths -- a function always returns.

Unfortunately, whilst conceptually simple it bloats code significantly -- many rare conditions have to either be explicitly handled or passed back at multiple points -- so whilst it obligates reliability for the diligent developer, the careless developer will festoon his or her code with empty handlers and whilst the resulting code will unquestionably "run", it will almost certainly do so uselessly and with no indication why it isn't working properly.

Thus, exceptions are the usual way of avoiding undue verbosity and unpleasant handle-errors-everywhere detail.

Therefore, I suggest:

  1. Exceptions exist, but all exceptions are checked exceptions.
  2. All possible exceptions must be identified in every function/procedure/method signature.
  3. All exceptions must either be handled at point of failure, or passed back up the call stack to be handled there.
  4. Empty exception handlers are not allowed, and every exception handler implicitly sends a notification -- to which user code may subscribe -- that the exception has been handled. This may be used to log, wrap, or rethrow any or all exceptions in the case where a lower level mechanism has otherwise handled an exception. I.e., any exceptions caught and handled at a low level can be intercepted by a higher level.

That mechanism must handle all possible failures except the runtime being so broken that exception handlers cannot run. In that case, do what typical JVMs do when the runtime fails catastrophically: log the error (typically to a file called hs_err_pid.log) with a rich set of details and die.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org
Quote from Dave Voorhis on June 13, 2021, 5:55 pm
Quote from dandl on June 13, 2021, 4:23 am
Quote from Dave Voorhis on June 12, 2021, 7:23 pm

With the current popular programming languages, we can almost achieve 'if it compiles it runs' but it requires developer effort -- the compiler will only obligate error handling in some places, not everywhere errors can occur -- and external dependencies have to be treated in either of two ways:

  1. Handle them like user input. I.e., assume the external facility is going to be wrong and/or broken most of the time and act accordingly. Of course, that requires careful design, and all constructs for handling external dependencies will be dynamic. E.g., lots of unpleasantly-verbose and error-prone row.getColumnValue("CustomerAddress") and the like, instead of row.CustomerAddress. Or, if row.CustomerAddress is supported by the language as a dynamic construct (sugared to look static) make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)
  2. Handle them like local dependencies. Create/generate/import static constructs, so you get to use row.CustomerAddress and it is genuinely static, but make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)

We could do better to obligate handling of possible failures -- or alternate code paths -- everywhere they could occur, in both local and external dependencies. That's the notion behind Java checked exceptions. That's why Java's standard database-access mechanism -- JDBC -- can throw a checked (i.e., must be explicitly caught somewhere) SQLException for every operation, and that's why higher-level operations based on JDBC (which is almost all of them) typically either can throw a checked SQLException, or throw some other checked exception that wraps a SQLException.

It works very well when done properly. I can write Java programs whose environment, both local and external, can fail and they'll degrade gracefully and helpfully inform -- via logs at least, and sometimes via helpful messages (where appropriate) to a designated user -- what's failed.

I accept this as a reasonable outline of the status quo. But please note: (a) these are strategies that have changed little in 40-odd years and (b) you can (for Java), but I can't (for Java) and you can't (for language X). It takes deep knowledge of the runtime environment (not just the language).

It's not so much that it takes deep knowledge of the runtime environment -- which, ideally, it shouldn't require -- but the fact that the failure mechanisms are often hidden and only appear as a runtime surprise rather than being explicit at compile-time, and in particular, without (as it should be) obligating the developer to consider them and handle them (even if badly, which may be better than nothing at all.)

I agree, mostly. The extra knowledge is needed to interpret error codes and devise strategies to resolve problems once revealed.

Where that approach goes wrong is when lazy developers create empty exception handling blocks or otherwise naively defeat the mechanism, and the program silently keeps running in a broken state, or crashes, or otherwise fails badly instead of helpfully (i.e., noisily) and gracefully degrading. But at least empty exception handling blocks are relatively easy to locate.

Or, it goes wrong when lazy or naive developers create libraries that should throw checked exceptions when external dependencies break (because they will), but instead throw unchecked (i.e., catching them is not required, and thus the program will crash if they've not caught) exceptions, which are notionally intended for things you can't possibly recover from, like out-of-memory or coding errors. Then you get runtime surprises and unhandled crashes, and that can be harder to fix because you often don't know the problem exists until you encounter it in production.

It goes wrong when a highly skilled and motivated developer tries to solve problems in new territory, without the time or inclination to first acquire all those new skills. It goes wrong because those strategies and underlying dependencies are not directly transferable but rely on skill and runtime code without static or pre-execution checking or testing. [It relies a lot on SO.]

What's an "SO"?

Stack Overflow.

So, how do we design systems to operate safely in the environment you describe within the range of a competent programmer not possessing requisite platform-specific skills? Assume the programmer is not to use any platform specifics such as exception handling. My suggestion:

  • Identify all the kinds of failure handling we might want to use (prevent execution, abort with log message; retry; substitute default data; substitute previous valid data; etc)
  • For each point of failure implement generic handlers for each kind (platform specific knowledge required for this step)
  • For each point of failure require the programmer to choose how it is to be handled (or default; or multiple) (language extension required)
  • Provide a mechanism to report on points of failure prior or during execution (without waiting until it gets hit).

My point: the new languages we need are those that will support safe execution: if it compiles, it runs.

Perhaps the conceptually simplest way is to make every function call return a value that is a union of its explicit return type and types representing all otherwise-unhandled failures, and all union types obligate code paths to handle all components of the union (though not necessarily each individually; default, other or similar is an acceptable case.)  In this scheme, there are no alternate code paths -- a function always returns.

Unfortunately, whilst conceptually simple it bloats code significantly -- many rare conditions have to either be explicitly handled or passed back at multiple points -- so whilst it obligates reliability for the diligent developer, the careless developer will festoon his or her code with empty handlers and whilst the resulting code will unquestionably "run", it will almost certainly do so uselessly and with no indication why it isn't working properly.

Thus, exceptions are the usual way of avoiding undue verbosity and unpleasant handle-errors-everywhere detail.

I dislike exceptions because they work like a goto. The handler has access to surrounding local state with no certainty what went before.It's really easy to produce really hard bugs. [Exception handlers can be made safe if they exist only at a global level for disaster recovery, but Java wasn't built like that.]

I strongly prefer the Command/Query paradigm, in which all Query calls are safe and Command calls return only success or failure (after which there is a safe Query to return more info). I think that was a feature of Eiffel. With that paradigm there is absolutely no need for exceptions, or the union/Maybe approach, but maybe the compiler should warn about failure to handle a false return.

Therefore, I suggest:

  1. Exceptions exist, but all exceptions are checked exceptions.
  2. All possible exceptions must be identified in every function/procedure/method signature.
  3. All exceptions must either be handled at point of failure, or passed back up the call stack to be handled there.
  4. Empty exception handlers are not allowed, and every exception handler implicitly sends a notification -- to which user code may subscribe -- that the exception has been handled. This may be used to log, wrap, or rethrow any or all exceptions in the case where a lower level mechanism has otherwise handled an exception. I.e., any exceptions caught and handled at a low level can be intercepted by a higher level.

That mechanism must handle all possible failures except the runtime being so broken that exception handlers cannot run. In that case, do what typical JVMs do when the runtime fails catastrophically: log the error (typically to a file called hs_err_pid.log) with a rich set of details and die.

No, that's just doubling down on the wrong answer. Exceptions out!

Andl - A New Database Language - andl.org
Quote from dandl on June 14, 2021, 2:27 am
Quote from Dave Voorhis on June 13, 2021, 5:55 pm
Quote from dandl on June 13, 2021, 4:23 am
Quote from Dave Voorhis on June 12, 2021, 7:23 pm

With the current popular programming languages, we can almost achieve 'if it compiles it runs' but it requires developer effort -- the compiler will only obligate error handling in some places, not everywhere errors can occur -- and external dependencies have to be treated in either of two ways:

  1. Handle them like user input. I.e., assume the external facility is going to be wrong and/or broken most of the time and act accordingly. Of course, that requires careful design, and all constructs for handling external dependencies will be dynamic. E.g., lots of unpleasantly-verbose and error-prone row.getColumnValue("CustomerAddress") and the like, instead of row.CustomerAddress. Or, if row.CustomerAddress is supported by the language as a dynamic construct (sugared to look static) make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)
  2. Handle them like local dependencies. Create/generate/import static constructs, so you get to use row.CustomerAddress and it is genuinely static, but make sure there's plenty of exception handling in anticipation of CustomerAddress vanishing unexpectedly. (Because it will.)

We could do better to obligate handling of possible failures -- or alternate code paths -- everywhere they could occur, in both local and external dependencies. That's the notion behind Java checked exceptions. That's why Java's standard database-access mechanism -- JDBC -- can throw a checked (i.e., must be explicitly caught somewhere) SQLException for every operation, and that's why higher-level operations based on JDBC (which is almost all of them) typically either can throw a checked SQLException, or throw some other checked exception that wraps a SQLException.

It works very well when done properly. I can write Java programs whose environment, both local and external, can fail and they'll degrade gracefully and helpfully inform -- via logs at least, and sometimes via helpful messages (where appropriate) to a designated user -- what's failed.

I accept this as a reasonable outline of the status quo. But please note: (a) these are strategies that have changed little in 40-odd years and (b) you can (for Java), but I can't (for Java) and you can't (for language X). It takes deep knowledge of the runtime environment (not just the language).

It's not so much that it takes deep knowledge of the runtime environment -- which, ideally, it shouldn't require -- but the fact that the failure mechanisms are often hidden and only appear as a runtime surprise rather than being explicit at compile-time, and in particular, without (as it should be) obligating the developer to consider them and handle them (even if badly, which may be better than nothing at all.)

I agree, mostly. The extra knowledge is needed to interpret error codes and devise strategies to resolve problems once revealed.

Where that approach goes wrong is when lazy developers create empty exception handling blocks or otherwise naively defeat the mechanism, and the program silently keeps running in a broken state, or crashes, or otherwise fails badly instead of helpfully (i.e., noisily) and gracefully degrading. But at least empty exception handling blocks are relatively easy to locate.

Or, it goes wrong when lazy or naive developers create libraries that should throw checked exceptions when external dependencies break (because they will), but instead throw unchecked (i.e., catching them is not required, and thus the program will crash if they've not caught) exceptions, which are notionally intended for things you can't possibly recover from, like out-of-memory or coding errors. Then you get runtime surprises and unhandled crashes, and that can be harder to fix because you often don't know the problem exists until you encounter it in production.

It goes wrong when a highly skilled and motivated developer tries to solve problems in new territory, without the time or inclination to first acquire all those new skills. It goes wrong because those strategies and underlying dependencies are not directly transferable but rely on skill and runtime code without static or pre-execution checking or testing. [It relies a lot on SO.]

What's an "SO"?

Stack Overflow.

So, how do we design systems to operate safely in the environment you describe within the range of a competent programmer not possessing requisite platform-specific skills? Assume the programmer is not to use any platform specifics such as exception handling. My suggestion:

  • Identify all the kinds of failure handling we might want to use (prevent execution, abort with log message; retry; substitute default data; substitute previous valid data; etc)
  • For each point of failure implement generic handlers for each kind (platform specific knowledge required for this step)
  • For each point of failure require the programmer to choose how it is to be handled (or default; or multiple) (language extension required)
  • Provide a mechanism to report on points of failure prior or during execution (without waiting until it gets hit).

My point: the new languages we need are those that will support safe execution: if it compiles, it runs.

Perhaps the conceptually simplest way is to make every function call return a value that is a union of its explicit return type and types representing all otherwise-unhandled failures, and all union types obligate code paths to handle all components of the union (though not necessarily each individually; default, other or similar is an acceptable case.)  In this scheme, there are no alternate code paths -- a function always returns.

Unfortunately, whilst conceptually simple it bloats code significantly -- many rare conditions have to either be explicitly handled or passed back at multiple points -- so whilst it obligates reliability for the diligent developer, the careless developer will festoon his or her code with empty handlers and whilst the resulting code will unquestionably "run", it will almost certainly do so uselessly and with no indication why it isn't working properly.

Thus, exceptions are the usual way of avoiding undue verbosity and unpleasant handle-errors-everywhere detail.

I dislike exceptions because they work like a goto. The handler has access to surrounding local state with no certainty what went before.It's really easy to produce really hard bugs. [Exception handlers can be made safe if they exist only at a global level for disaster recovery, but Java wasn't built like that.]

I strongly prefer the Command/Query paradigm, in which all Query calls are safe and Command calls return only success or failure (after which there is a safe Query to return more info). I think that was a feature of Eiffel. With that paradigm there is absolutely no need for exceptions, or the union/Maybe approach, but maybe the compiler should warn about failure to handle a false return.

That's the same approach as my all-functions-return-a-union, only instead of the 'command' invocation returning the 'query' value, you've got two calls instead of one.

That still has the problem of needing to deal with each failure individually. Do you really want to do this?

var nameResult = row.Name;
if (nameResult.isError()) {
  ... handle Name error ...
  return Operation.Failure.NAME;
}
var name = nameResult.getValue();
var addressResult = row.Address;
if (addressResult.isError()) {
  ... handle Address error ...
  return Operation.Failure.ADDRESS;
}
var address = addressResult.getValue();

... etc ...

Tedious.

I'd rather just do this:

try {
  var name = row.Name;
  var address = row.Address;
} catch (Exception e) {
  return Operation.Failure.STAFF_ROWSET;
}

I appreciate that you -- and others -- don't like the introduced scope and associated issues of a try { } catch block. That's fine -- there are other ways of defining exception handlers. Some are more goto-like -- like VB6 and older exception handling (i.e., awful) -- and some are less goto-like, such as defining exception handling at a function/procedure/method level so that a "caught" exception is effectively akin to an immediate return.

There are pros and cons -- and thus inevitable tradeoffs -- with these and other approaches.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org

I strongly prefer the Command/Query paradigm, in which all Query calls are safe and Command calls return only success or failure (after which there is a safe Query to return more info). I think that was a feature of Eiffel. With that paradigm there is absolutely no need for exceptions, or the union/Maybe approach, but maybe the compiler should warn about failure to handle a false return.

That's the same approach as my all-functions-return-a-union, only instead of the 'command' invocation returning the 'query' value, you've got two calls instead of one.

Usually not. It might help if you read up on CQS: https://en.wikipedia.org/wiki/Command%E2%80%93query_separation, or the Eiffel book by Bertrand Meyer.

That still has the problem of needing to deal with each failure individually. Do you really want to do this?

var nameResult = row.Name;
if (nameResult.isError()) {
... handle Name error ...
return Operation.Failure.NAME;
}
var name = nameResult.getValue();
var addressResult = row.Address;
if (addressResult.isError()) {
... handle Address error ...
return Operation.Failure.ADDRESS;
}
var address = addressResult.getValue();
... etc ...
var nameResult = row.Name; if (nameResult.isError()) { ... handle Name error ... return Operation.Failure.NAME; } var name = nameResult.getValue(); var addressResult = row.Address; if (addressResult.isError()) { ... handle Address error ... return Operation.Failure.ADDRESS; } var address = addressResult.getValue(); ... etc ...
var nameResult = row.Name;
if (nameResult.isError()) {
  ... handle Name error ...
  return Operation.Failure.NAME;
}
var name = nameResult.getValue();
var addressResult = row.Address;
if (addressResult.isError()) {
  ... handle Address error ...
  return Operation.Failure.ADDRESS;
}
var address = addressResult.getValue();

... etc ...

Tedious.

And totally wrong.

I'd rather just do this:

try {
var name = row.Name;
var address = row.Address;
} catch (Exception e) {
return Operation.Failure.STAFF_ROWSET;
}
try { var name = row.Name; var address = row.Address; } catch (Exception e) { return Operation.Failure.STAFF_ROWSET; }
try {
  var name = row.Name;
  var address = row.Address;
} catch (Exception e) {
  return Operation.Failure.STAFF_ROWSET;
}

In CQS it looks like this:

// read is a command and may fail, get is a query and is always safe
if (!row.readStaffRowset()) return row.getErrorInfo();
// now use row.getName() and row.getAddress() safely.

In cases likes this, CQS is usually shorter.

I appreciate that you -- and others -- don't like the introduced scope and associated issues of a try { } catch block. That's fine -- there are other ways of defining exception handlers. Some are more goto-like -- like VB6 and older exception handling (i.e., awful) -- and some are less goto-like, such as defining exception handling at a function/procedure/method level so that a "caught" exception is effectively akin to an immediate return.

There are pros and cons -- and thus inevitable tradeoffs -- with these and other approaches.

There are no good exceptions other than those that completely unwind to an outer state that is known to be valid, for disaster recovery. CQS is usually the right choice and suffers none of these problems.

Andl - A New Database Language - andl.org
Quote from dandl on June 14, 2021, 2:19 pm

I strongly prefer the Command/Query paradigm, in which all Query calls are safe and Command calls return only success or failure (after which there is a safe Query to return more info). I think that was a feature of Eiffel. With that paradigm there is absolutely no need for exceptions, or the union/Maybe approach, but maybe the compiler should warn about failure to handle a false return.

That's the same approach as my all-functions-return-a-union, only instead of the 'command' invocation returning the 'query' value, you've got two calls instead of one.

Usually not. It might help if you read up on CQS: https://en.wikipedia.org/wiki/Command%E2%80%93query_separation, or the Eiffel book by Bertrand Meyer.

That still has the problem of needing to deal with each failure individually. Do you really want to do this?

var nameResult = row.Name;
if (nameResult.isError()) {
... handle Name error ...
return Operation.Failure.NAME;
}
var name = nameResult.getValue();
var addressResult = row.Address;
if (addressResult.isError()) {
... handle Address error ...
return Operation.Failure.ADDRESS;
}
var address = addressResult.getValue();
... etc ...
var nameResult = row.Name; if (nameResult.isError()) { ... handle Name error ... return Operation.Failure.NAME; } var name = nameResult.getValue(); var addressResult = row.Address; if (addressResult.isError()) { ... handle Address error ... return Operation.Failure.ADDRESS; } var address = addressResult.getValue(); ... etc ...
var nameResult = row.Name;
if (nameResult.isError()) {
  ... handle Name error ...
  return Operation.Failure.NAME;
}
var name = nameResult.getValue();
var addressResult = row.Address;
if (addressResult.isError()) {
  ... handle Address error ...
  return Operation.Failure.ADDRESS;
}
var address = addressResult.getValue();

... etc ...

Tedious.

And totally wrong.

No, it's exactly what using Command/Query for error handling implies. I am, of course, assuming row.Name and row.Address each potentially produce an error condition that we wish to consider individually.

I'd rather just do this:

try {
var name = row.Name;
var address = row.Address;
} catch (Exception e) {
return Operation.Failure.STAFF_ROWSET;
}
try { var name = row.Name; var address = row.Address; } catch (Exception e) { return Operation.Failure.STAFF_ROWSET; }
try {
  var name = row.Name;
  var address = row.Address;
} catch (Exception e) {
  return Operation.Failure.STAFF_ROWSET;
}

In CQS it looks like this:

// read is a command and may fail, get is a query and is always safe
if (!row.readStaffRowset()) return row.getErrorInfo();
// now use row.getName() and row.getAddress() safely.

In cases likes this, CQS is usually shorter.

It's not the same example. If we can handle all possible errors in readStaffRowset such that we can presume row.getName() and row.getAddress() will always succeed thereafter, that's a different scenario. My illustration was intended to represent a series of operations where each can individually generate errors. I picked row.Name and row.Address arbitrarily; they could be anything.

In other words, assume the problem is that command/query obligates this:

if (!thing1.someOperation1()) 
   return new Error(thing1.someError1());
if (!thing2.someOp2()) 
   return new Error(thing2.someError2());
if (!thing3, someOpn3())
   return new Error(thing3.someError3());

I'd rather it were this:

try {
   thing1.someOperation1();
   thing2.someOp2();
   thing3.someOpn3();
   return null;
} catch (Exception e) {
   return e;
}

Or similar. Again, there are various ways of specifying exception handling (with various tradeoffs) and it also may be desirable to intercept low level handling at higher level. My main issue with exception handling in C# and Java is that exceptions can be thrown by a method without knowing that they'll be thrown, but the developer who wrote the method (or the developer who wrote the methods it uses, and so on) knows. That the caller doesn't know what can be thrown should be impossible.

Thus, all exceptions should either be checked exceptions or -- in the case of "system" exceptions like running out of memory or stack that can essentially occur anywhere (and everywhere) -- there should be an obligatory root-level handler. (The Java platform allows you to specify such a handler, but it's optional.)

I appreciate that you -- and others -- don't like the introduced scope and associated issues of a try { } catch block. That's fine -- there are other ways of defining exception handlers. Some are more goto-like -- like VB6 and older exception handling (i.e., awful) -- and some are less goto-like, such as defining exception handling at a function/procedure/method level so that a "caught" exception is effectively akin to an immediate return.

There are pros and cons -- and thus inevitable tradeoffs -- with these and other approaches.

There are no good exceptions other than those that completely unwind to an outer state that is known to be valid, for disaster recovery. CQS is usually the right choice and suffers none of these problems.

It can't be easily intercepted, it's potentially very laborious, and it's easy to miss out handling that should be obligatory.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org
Quote from Dave Voorhis on June 14, 2021, 3:51 pm
Quote from dandl on June 14, 2021, 2:19 pm

I strongly prefer the Command/Query paradigm, in which all Query calls are safe and Command calls return only success or failure (after which there is a safe Query to return more info). I think that was a feature of Eiffel. With that paradigm there is absolutely no need for exceptions, or the union/Maybe approach, but maybe the compiler should warn about failure to handle a false return.

That's the same approach as my all-functions-return-a-union, only instead of the 'command' invocation returning the 'query' value, you've got two calls instead of one.

Usually not. It might help if you read up on CQS: https://en.wikipedia.org/wiki/Command%E2%80%93query_separation, or the Eiffel book by Bertrand Meyer.

That still has the problem of needing to deal with each failure individually. Do you really want to do this?

var nameResult = row.Name;
if (nameResult.isError()) {
... handle Name error ...
return Operation.Failure.NAME;
}
var name = nameResult.getValue();
var addressResult = row.Address;
if (addressResult.isError()) {
... handle Address error ...
return Operation.Failure.ADDRESS;
}
var address = addressResult.getValue();
... etc ...
var nameResult = row.Name; if (nameResult.isError()) { ... handle Name error ... return Operation.Failure.NAME; } var name = nameResult.getValue(); var addressResult = row.Address; if (addressResult.isError()) { ... handle Address error ... return Operation.Failure.ADDRESS; } var address = addressResult.getValue(); ... etc ...
var nameResult = row.Name;
if (nameResult.isError()) {
  ... handle Name error ...
  return Operation.Failure.NAME;
}
var name = nameResult.getValue();
var addressResult = row.Address;
if (addressResult.isError()) {
  ... handle Address error ...
  return Operation.Failure.ADDRESS;
}
var address = addressResult.getValue();

... etc ...

Tedious.

And totally wrong.

No, it's exactly what using Command/Query for error handling implies. I am, of course, assuming row.Name and row.Address each potentially produce an error condition that we wish to consider individually.

No, it's exactly and totally wrong. A command returns success or failure and no data; a query returns data and never fails.

I'd rather just do this:

try {
var name = row.Name;
var address = row.Address;
} catch (Exception e) {
return Operation.Failure.STAFF_ROWSET;
}
try { var name = row.Name; var address = row.Address; } catch (Exception e) { return Operation.Failure.STAFF_ROWSET; }
try {
  var name = row.Name;
  var address = row.Address;
} catch (Exception e) {
  return Operation.Failure.STAFF_ROWSET;
}

In CQS it looks like this:

// read is a command and may fail, get is a query and is always safe
if (!row.readStaffRowset()) return row.getErrorInfo();
// now use row.getName() and row.getAddress() safely.

In cases likes this, CQS is usually shorter.

It's not the same example. If we can handle all possible errors in readStaffRowset such that we can presume row.getName() and row.getAddress() will always succeed thereafter, that's a different scenario. My illustration was intended to represent a series of operations where each can individually generate errors. I picked row.Name and row.Address arbitrarily; they could be anything.

In other words, assume the problem is that command/query obligates this:

if (!thing1.someOperation1())
return new Error(thing1.someError1());
if (!thing2.someOp2())
return new Error(thing2.someError2());
if (!thing3, someOpn3())
return new Error(thing3.someError3());
if (!thing1.someOperation1()) return new Error(thing1.someError1()); if (!thing2.someOp2()) return new Error(thing2.someError2()); if (!thing3, someOpn3()) return new Error(thing3.someError3());
if (!thing1.someOperation1()) 
   return new Error(thing1.someError1());
if (!thing2.someOp2()) 
   return new Error(thing2.someError2());
if (!thing3, someOpn3())
   return new Error(thing3.someError3());

I'd rather it were this:

try {
thing1.someOperation1();
thing2.someOp2();
thing3.someOpn3();
return null;
} catch (Exception e) {
return e;
}
try { thing1.someOperation1(); thing2.someOp2(); thing3.someOpn3(); return null; } catch (Exception e) { return e; }
try {
   thing1.someOperation1();
   thing2.someOp2();
   thing3.someOpn3();
   return null;
} catch (Exception e) {
   return e;
}

You're forcing the code into a mould to satisfy a prejudice, and now you've left out all the queries and you still haven't reported the error. But we can write that in CQS like this:

if (thing1.someOperation1() && thing2.someOp2() && thing3.someOpn3()) return true;
log.writeError(thing1.getError());
log.writeError(thing2.getError());
log.writeError(thing3.getError());
return false;

In typical usage CQS is about the same or slightly shorter than exception handling, and always safer.

 

Or similar. Again, there are various ways of specifying exception handling (with various tradeoffs) and it also may be desirable to intercept low level handling at higher level. My main issue with exception handling in C# and Java is that exceptions can be thrown by a method without knowing that they'll be thrown, but the developer who wrote the method (or the developer who wrote the methods it uses, and so on) knows. That the caller doesn't know what can be thrown should be impossible.

Thus, all exceptions should either be checked exceptions or -- in the case of "system" exceptions like running out of memory or stack that can essentially occur anywhere (and everywhere) -- there should be an obligatory root-level handler. (The Java platform allows you to specify such a handler, but it's optional.)

I appreciate that you -- and others -- don't like the introduced scope and associated issues of a try { } catch block. That's fine -- there are other ways of defining exception handlers. Some are more goto-like -- like VB6 and older exception handling (i.e., awful) -- and some are less goto-like, such as defining exception handling at a function/procedure/method level so that a "caught" exception is effectively akin to an immediate return.

There are pros and cons -- and thus inevitable tradeoffs -- with these and other approaches.

There are no good exceptions other than those that completely unwind to an outer state that is known to be valid, for disaster recovery. CQS is usually the right choice and suffers none of these problems.

It can't be easily intercepted, it's potentially very laborious, and it's easy to miss out handling that should be obligatory.

You lost me. Most languages do not force any kind of exception handling, Java is the odd man out. I'm talking about new language features to achieve safety, and CQS with language amd library support is what I propose. It's shorter and safer, easier to reason about, and you can still have global exceptions for 'get me out of here now!'. Why do you object?

Andl - A New Database Language - andl.org
Quote from dandl on June 15, 2021, 12:38 am
Quote from Dave Voorhis on June 14, 2021, 3:51 pm
Quote from dandl on June 14, 2021, 2:19 pm

I strongly prefer the Command/Query paradigm, in which all Query calls are safe and Command calls return only success or failure (after which there is a safe Query to return more info). I think that was a feature of Eiffel. With that paradigm there is absolutely no need for exceptions, or the union/Maybe approach, but maybe the compiler should warn about failure to handle a false return.

That's the same approach as my all-functions-return-a-union, only instead of the 'command' invocation returning the 'query' value, you've got two calls instead of one.

Usually not. It might help if you read up on CQS: https://en.wikipedia.org/wiki/Command%E2%80%93query_separation, or the Eiffel book by Bertrand Meyer.

That still has the problem of needing to deal with each failure individually. Do you really want to do this?

var nameResult = row.Name;
if (nameResult.isError()) {
... handle Name error ...
return Operation.Failure.NAME;
}
var name = nameResult.getValue();
var addressResult = row.Address;
if (addressResult.isError()) {
... handle Address error ...
return Operation.Failure.ADDRESS;
}
var address = addressResult.getValue();
... etc ...
var nameResult = row.Name; if (nameResult.isError()) { ... handle Name error ... return Operation.Failure.NAME; } var name = nameResult.getValue(); var addressResult = row.Address; if (addressResult.isError()) { ... handle Address error ... return Operation.Failure.ADDRESS; } var address = addressResult.getValue(); ... etc ...
var nameResult = row.Name;
if (nameResult.isError()) {
  ... handle Name error ...
  return Operation.Failure.NAME;
}
var name = nameResult.getValue();
var addressResult = row.Address;
if (addressResult.isError()) {
  ... handle Address error ...
  return Operation.Failure.ADDRESS;
}
var address = addressResult.getValue();

... etc ...

Tedious.

And totally wrong.

No, it's exactly what using Command/Query for error handling implies. I am, of course, assuming row.Name and row.Address each potentially produce an error condition that we wish to consider individually.

No, it's exactly and totally wrong. A command returns success or failure and no data; a query returns data and never fails.

I'd rather just do this:

try {
var name = row.Name;
var address = row.Address;
} catch (Exception e) {
return Operation.Failure.STAFF_ROWSET;
}
try { var name = row.Name; var address = row.Address; } catch (Exception e) { return Operation.Failure.STAFF_ROWSET; }
try {
  var name = row.Name;
  var address = row.Address;
} catch (Exception e) {
  return Operation.Failure.STAFF_ROWSET;
}

In CQS it looks like this:

// read is a command and may fail, get is a query and is always safe
if (!row.readStaffRowset()) return row.getErrorInfo();
// now use row.getName() and row.getAddress() safely.

In cases likes this, CQS is usually shorter.

It's not the same example. If we can handle all possible errors in readStaffRowset such that we can presume row.getName() and row.getAddress() will always succeed thereafter, that's a different scenario. My illustration was intended to represent a series of operations where each can individually generate errors. I picked row.Name and row.Address arbitrarily; they could be anything.

In other words, assume the problem is that command/query obligates this:

if (!thing1.someOperation1())
return new Error(thing1.someError1());
if (!thing2.someOp2())
return new Error(thing2.someError2());
if (!thing3, someOpn3())
return new Error(thing3.someError3());
if (!thing1.someOperation1()) return new Error(thing1.someError1()); if (!thing2.someOp2()) return new Error(thing2.someError2()); if (!thing3, someOpn3()) return new Error(thing3.someError3());
if (!thing1.someOperation1()) 
   return new Error(thing1.someError1());
if (!thing2.someOp2()) 
   return new Error(thing2.someError2());
if (!thing3, someOpn3())
   return new Error(thing3.someError3());

I'd rather it were this:

try {
thing1.someOperation1();
thing2.someOp2();
thing3.someOpn3();
return null;
} catch (Exception e) {
return e;
}
try { thing1.someOperation1(); thing2.someOp2(); thing3.someOpn3(); return null; } catch (Exception e) { return e; }
try {
   thing1.someOperation1();
   thing2.someOp2();
   thing3.someOpn3();
   return null;
} catch (Exception e) {
   return e;
}

You're forcing the code into a mould to satisfy a prejudice, and now you've left out all the queries and you still haven't reported the error. But we can write that in CQS like this:

if (thing1.someOperation1() && thing2.someOp2() && thing3.someOpn3()) return true;
log.writeError(thing1.getError());
log.writeError(thing2.getError());
log.writeError(thing3.getError());
return false;

In typical usage CQS is about the same or slightly shorter than exception handling, and always safer.

 

Or similar. Again, there are various ways of specifying exception handling (with various tradeoffs) and it also may be desirable to intercept low level handling at higher level. My main issue with exception handling in C# and Java is that exceptions can be thrown by a method without knowing that they'll be thrown, but the developer who wrote the method (or the developer who wrote the methods it uses, and so on) knows. That the caller doesn't know what can be thrown should be impossible.

Thus, all exceptions should either be checked exceptions or -- in the case of "system" exceptions like running out of memory or stack that can essentially occur anywhere (and everywhere) -- there should be an obligatory root-level handler. (The Java platform allows you to specify such a handler, but it's optional.)

I appreciate that you -- and others -- don't like the introduced scope and associated issues of a try { } catch block. That's fine -- there are other ways of defining exception handlers. Some are more goto-like -- like VB6 and older exception handling (i.e., awful) -- and some are less goto-like, such as defining exception handling at a function/procedure/method level so that a "caught" exception is effectively akin to an immediate return.

There are pros and cons -- and thus inevitable tradeoffs -- with these and other approaches.

There are no good exceptions other than those that completely unwind to an outer state that is known to be valid, for disaster recovery. CQS is usually the right choice and suffers none of these problems.

It can't be easily intercepted, it's potentially very laborious, and it's easy to miss out handling that should be obligatory.

You lost me. Most languages do not force any kind of exception handling, Java is the odd man out. I'm talking about new language features to achieve safety, and CQS with language amd library support is what I propose. It's shorter and safer, easier to reason about, and you can still have global exceptions for 'get me out of here now!'. Why do you object?

Exactly, most languages do not force any kind of exception handling and that is the problem.

Languages should force exception handling, because exceptions, no matter how rare, will occur.

Java was on the right track with checked exceptions, but it didn't go far enough. It should have made (almost) all exceptions be checked exceptions.

Instead, it distinguished checked (must handle; designed for recoverable exceptions, often in external dependencies) from unchecked (no need to handle; mainly intended for programming errors or irrecoverable like out-of-memory) exceptions. It was done with good intent -- to address precisely the sort of problem we're talking about here -- but the inevitable result is that methods exist that can throw unchecked exceptions without the user of those methods knowing it can happen, so code gets written that can fail in an uncontrolled fashion.

So if you want to write programs that keeping running no matter what (if it compiles, it runs), you make it a practice to wrap everything in general exception handlers. If something throws an exception, any exception, you catch it and degrade gracefully or restart after a delay or whatever.  That works, but it could be better.

The problem is still that nothing obligates the developer to implement such mechanisms, and there's no (easy) way for higher levels to intercept exceptions thrown and caught inside lower-level mechanisms.

Thus, all exceptions should be checked exceptions, and in the case of "system" exceptions like running out of memory or stack that can occur everywhere, there should be an obligatory root-level handler.

Command-and-query is fine for "happy path" code, but it's not really an exception handling mechanism. There is nothing in it to obligate handling exceptions, so it encourages writing broken code that does nothing but issue commands and fail silently. That's not resilience and safety, that's just quiet breakage.

But if it had mechanisms that obligate checking status after issuing commands, then it would be better. It would still tediously require an explicit check at every 'command' rather than concisely wrapping a collection of related commands in a single any-failure-in-these-should-do-this exception handler, but at least it would obligate dealing with exceptions, and it still needs a mechanism to optionally intercept low-level exception handling at higher levels.

I'm the forum administrator and lead developer of Rel. Email me at dave@armchair.mb.ca with the Subject 'TTM Forum'. Download Rel from https://reldb.org
PreviousPage 2 of 3Next