Metaethics asks "What kind of stuff is goodness made of?" (or "How would we compute goodness?") rather than "Which particular policies or outcomes are good or not-good?" (In other words, it asks meta-level questions about ethics, rather than object-level moral questions like, "Ought I to steal from banks in order to give the money to the deserving poor?")
Metaethics is one of the three branches of ethics usually recognized by philosophers, the others being normative ethics and applied ethics.
Eliezer Yudkowsky wrote a Sequence about metaethics, the Metaethics sequence, which Yudkowsky worried failed to convey his central point (this post by Luke tried to clarify); he approached the same problem again from a different angle in Highly Advanced Epistemology 101 for Beginners. From a standard philosophical standpoint, Yudkowsky's philosophy is closest to Frank Jackson's moral functionalism / analytic descriptivism; Yudkowsky could be loosely characterized as moral cognitivist - someone who believes moral sentences are either true or false - but not a moral realist - thus denying that moral sentences refer to facts about the world. Yudkowsky believes that moral cognition in any single human is at least potentially about a subject matter that is 'logical' in the sense that its semantics can be pinned down by axioms, and hence that moral cognition can bear truth-values; also that human beings both using similar words like "morality" can be talking about highly overlapping subject matter; but not that all possible minds would find the truths about this subject matter to be psychologically compelling.
Luke Muehlhauser has written a sequence, No-Nonsense Metaethics, where he claims that many of the questions of metaethics can be answered today using modern neuroscience and rationality. He explains how conventional metaethics or "Austere Metaethics" is capable of, after assuming a definition of 'right', choosing the right action given a situation - but useless without assuming some criteria for 'right'. He proposes instead "Empathic Metaethics" which utilizes your underlying cognitive algorithms to understand what you think 'right' means, helps clarify any emotional and cognitive contradictions in it, and then tells you what the right thing to do is, according to your definition of right. This approach is highly relevant for the Friendly AI problem as a way of defining human-like goals and motivations when designing AIs.