Consider reading:

• Editing advice for LW users
• Style: Ten Lessons in Clarity and Grace (and doing the exercises)

For editing grammar, passing through GPT-4 can be pretty good, but you have to be very clear in your prompting to get it to change nothing else.

(Somewhat obvious advice, but hopefully useful.)

Paul Graham, one of the best technical writers I know, wrote an article called "Writing, Briefly". I'll quote it in full.

(In the process of answering an email, I accidentally wrote a tiny essay about writing. I usually spend weeks on an essay. This one took 67 minutes—23 of writing, and 44 of rewriting.)

I think it's far more important to write well than most people realize. Writing doesn't just communicate ideas; it generates them. If you're bad at writing and don't like to do it, you'll miss out on most of the ideas writing would have generated.

As for how to write well, here's the short version: Write a bad version 1 as fast as you can; rewrite it over and over; cut out everything unnecessary; write in a conversational tone; develop a nose for bad writing, so you can see and fix it in yours; imitate writers you like; if you can't get started, tell someone what you plan to write about, then write down what you said; expect 80% of the ideas in an essay to happen after you start writing it, and 50% of those you start with to be wrong; be confident enough to cut; have friends you trust read your stuff and tell you which bits are confusing or drag; don't (always) make detailed outlines; mull ideas over for a few days before writing; carry a small notebook or scrap paper with you; start writing when you think of the first sentence; if a deadline forces you to start before that, just say the most important sentence first; write about stuff you like; don't try to sound impressive; don't hesitate to change the topic on the fly; use footnotes to contain digressions; use anaphora to knit sentences together; read your essays out loud to see (a) where you stumble over awkward phrases and (b) which bits are boring (the paragraphs you dread reading); try to tell the reader something new and useful; work in fairly big quanta of time; when you restart, begin by rereading what you have so far; when you finish, leave yourself something easy to start with; accumulate notes for topics you plan to cover at the bottom of the file; don't feel obliged to cover any of them; write for a reader who won't read the essay as carefully as you do, just as pop songs are designed to sound ok on crappy car radios; if you say anything mistaken, fix it immediately; ask friends which sentence you'll regret most; go back and tone down harsh remarks; publish stuff online, because an audience makes you write more, and thus generate more ideas; print out drafts instead of just looking at them on the screen; use simple, germanic words; learn to distinguish surprises from digressions; learn to recognize the approach of an ending, and when one appears, grab it.

I've landed three jobs thanks to my writing, which is a bit strange to say because I still feel like I have substantial room for improvement. But that's still a pretty good track record, so I'll tell you what has worked for me:

• Ask people to read your stuff and tell you at what point they get bored or want to stop. Tell them to be brutally honest. The most important part of writing is keeping your audience's attention, and many writers make no effort to do this.
• Write about something that's actually important, and that interests you. I've done a large amount of high-quality writing about interest rates, banking and crypto. So far as I can tell it was a complete waste because no one cared. 80% of the battle is just picking the right topic.
• Put the most imortant ideas at the start of whatever you're writing. The drop-off among readers (even on a site like LessWrong) is shockingly high. I received almost 400 upvotes on my post about adult intelligence enhancement, and only four people sent me a DM in response to my request they do so that I placed at the end of the appendix.
• Write the narrative of a story in the evening and then correct language and facts in the morning. My best, most productive narrative writing often occurs late at night, but when I re-read it in the morning it sounds sloppy and full of mistakes. However, I nearly always need to start with the sloppy, emotional version because good narrative writing is the one thing I can't do well when my brain is functioning at full capacity.
• Don't be afraid to start a new draft of a post if you feel like you haven't gotten the narrative right.

The first lecture at this link and accompanying handout from UChicago's (now-retired) writing-program director, Larry McEnerney, has come up here on a number of occasions.

Additionally, I imagine you'll be able to unearth some good stuff perusing LW's writing tag.

Have AI text-to-speech read your work back to you. I catch a lot of tense mismatches and bad phrasings this way. 

This advice isn't specific to writing, but I believe it still applies.  To get better at something, it's very often necessary to stop worrying about the overall thing and try to get better at components and contributing skills.

For technical or idea-communicative pieces, you can explicitly work on sentence length, paragraph relevance (what are you actually communicating with each fragment?), the very hard skills of writing for diverse reader preferences, etc. 

Also applicable to many domains, cultivate feedback avenues.  Getting someone to tell you what they got out of a piece, and what felt like it interfered, is absolute gold.  But somewhat difficult to find if you're not part of communities where it's common.

Edit your writing. If you have the self-awareness that your sentences are too long, edit them to be shorter. 

Asking GPT-4 to critique your writing and dialog about ways to improve it is also useful.

Just speculating here, but I suspect that becoming a good writer comes in two steps. First, you must get from actively bad to neutral. Second, you get from neutral to good.

I have no idea how to do the second part. I think I can advise you on the first, because although the great writers have their unique voices, the bad writers typically keep making the same mistakes. (Trying to do the second part before mastering the first part results in bad writing. You need to get rid of the bad habits first.) This is the first part:

Most importantly, write shortly and to the point. Text can be shortened on multiple levels: sentences contain extra words; articles contain extra sections. Does the meaning of the sentence change substantially if you skip this word? If not, skip it. Would the article still make its intended point without this section? If yes, remove it. (If it breaks your heart, maybe move it to a footnote, or decide to keep it for a follow-up article.)

An example of a section that can be removed is introduction. Instead of "in this article, I will tell you what X is", simply start by describing X. (Generally, meta goes last, if at all.)

Use simple words rather than complicated ones, if the meaning is the same. Short sentences rather than long ones. Don't make the text needlessly costly for the reader.

Group sentences to paragraphs (one thought per paragraph), and paragraphs to sections. Makes the text easier to navigate. (Too long text -- insert pictures.)

In fiction, I would say: use sensory words. Describe what the characters see or hear or feel, not just their opinions or thoughts. In non-fiction, specific examples are better than abstractions (give examples first, generalize later), and illustrations can help.

Don't try to impress. That usually backfires. Anything "artistic" or "high-status" probably just sucks, e.g. modern journalism.

If it is possible to explain to a child, do it that way. Your task is to transmit thoughts, not signal your sophistication. Impress by concepts, not by choice of words.

If you use equations, explain what the individual symbols mean. Even mathematicians may not necessarily be familiar with conventions in your branch of math.

(In the second part you go against some of this advice, e.g. you insert jokes even if they are not strictly necessary, you use complicated words as a kind of insider jokes, etc. But the difference is that you only do that on purpose and at the right place, not habitually.)

The advice I wish I had earlier is:

• https://dilbertblog.typepad.com/the_dilbert_blog/2007/06/the_day_you_bec.html
• read one of Derek Sivers's books (as an example)
• Derek Sivers also has some recommendations for some writing books that are probably good. I tried Several Short Sentences About Writing and idk it was okay for me, some people really like it though

LW is full of good writers

I disagree with this definitively. I can't read most if not almost all LW posts. You can do much better if you know what you want to communicate and only say the essential words. 

I second GeneSmith’s suggestion to ask readers for feedback. Be aware that this is something of an imposition and that you’re asking people to spend time and energy critiquing what is currently not great writing. If possible, offer to trade - find some other people with similar problems and offer to critique their writing. For fiction, you can do this on CritiqueCircle but I don’t know of an organised equivalent for non-fiction.

The other thing you can do is to iterate. When you write something, say to yourself that you are writing the first draft of X. Then go away and do something else, come back to your writing later, and ask how you can edit it to make it better. You already described problems like using too many long sentences. So edit your work to remove them. If possible, aim to edit the day after writing - it helps if you can sleep on it. If you have time constraints, at least go away and get a cup of coffee or something in order to separate writing time from editing time.

For essays:

1. Write a short outline and then do lots of research. 
2. Using the outline and your research have a long conversation about the topic with a person that you're used to having long conversations with. If it helps you can record the conversation or take notes.
3. Write the essay the morning or night after the conversation. With essays I find it's better to work in small bursts (20-60 minutes) and go back to it periodically but that may just be a personal preference.
4. Have someone read it.
5. Edit.

For narratives:

1. Write it either in bursts or one sitting and don't think about it too much. You can do it by hand or typed but don't be afraid to jump around, write in the margins or write footnotes to use later or articulate what you're trying to say or effect you want to achieve without including in the actual narrative. Just keep going until you feel satisfied or feel like you're not making progress.
2. Put it away and don't look at it or think about it for a period of time (2 weeks to 6 months).
3. Reread and edit.
4. Have someone read it.
5. Edit.

As a professional novelist, the best advice I can give comes from one of the greatest writers of the 20th century, Ernest Hemingway: "The first draft of anything is shit." He was known to rewrite his short stories up to 30 times. So, rewrite. It helps to let some time pass (at least a few days) before you reread and rewrite a text. This makes it easier to spot the weak parts.

For me, rewriting often means cutting things out that aren't really necessary. That hurts, because I have put some effort into putting the words there in the first place. So I use a simple trick to overcome my reluctance: I don't just delete the text, but cut it out and copy it into a seperate document for each novel, called "cutouts". That way, I can always reverse my decision to cut things out or maybe reuse parts later, and I don't have the feeling that the work is "lost". Of course, I rarely reuse those cutouts. 

I also agree with the other answers regarding reader feedback, short sentences, etc. All of this is part of the rewriting process.

exceedingly long and complex sentences

Break them down. Long sentences in a comfortable cadence like being punctuated by short ones.

giving masses of detail with little apparent regard for how much information the person at the other end actually needs

Give more regard to what the reader needs.

"stick-on" weird metaphors which appear randomly every time I’m afraid I’m being too technical or annoying (so you get a wall of annoying text with a bit of canned laughter in the middle…)

Have you asked readers whether they dislike the metaphors?

vague sentences that go around for a while as I’m slowly figuring out what I mean to say

Rewrite after discovering your own intent. That's what editing is for.

long paragraphs, etc.

Fortunately your keyboard has an enter key, with no limit of uses.

Also, I spend ages proofreading anything I write and worrying about it…

Write where you feel that the stakes are low. If the consequences of poor proofreading don't feel worth worrying about, you can practice the skill of worrying less.