There are all sorts of different inline tags, and I haven't found a place that explains them all in one place, so here is the quick and dirty...

<% ... %>

The most basic inline tag, basically runs normal code: 

<% if (User.IsInRole("admin")) { %>
  You can see this
<% } else { %>
  You are no admin fool!
<%} %>

http://msdn2.microsoft.com/en-us/library/ms178135(vs.80).aspx

<%= ... %>

Used for small chunks of information, usually from objects and single pieces of information like a single string or int variable: 

The Date is now <%= DateTime.Now.ToShortDateString() %>
The value of string1 is <%= string1 %> 

http://msdn2.microsoft.com/en-us/library/6dwsdcf5(VS.71).aspx

*note: <%= is the equivalent of Response.Write() -  Courtesy of Adam from the US,thanks!

<%# .. %>

Used for Binding Expressions; such as Eval and Bind, most often found in data controls like GridView, Repeater, etc.:

<asp:Repeater ID="rptMeetings" DataSourceID="meetings" runat="server">
    <ItemTemplate>
        <%# Eval("MeetingName") %>
    </ItemTemplate>
</asp:Repeater>
  

http://msdn2.microsoft.com/en-us/library/ms178366.aspx

<%$ ... %>

Used for expressions, not code; often seen with DataSources: 

<asp:SqlDataSource ID="party" runat="server" ConnectionString="<%$ ConnectionStrings:letsParty %>" SelectCommand="SELECT * FROM [table]" />

http://msdn2.microsoft.com/en-us/library/d5bd1tad.aspx

<%@ ... %>

This is for directive syntax; basically the stuff you see at the top your your aspx pages like control registration and page declaration: 

<%@ Page Language="C#" MasterPageFile="~/MasterPage.master" AutoEventWireup="true" CodeFile="Default.aspx.cs" Inherits="_Default" Title="Untitled Page" %>
<%@ Register TagPrefix="wp" Namespace="CustomWebParts" %>

http://msdn2.microsoft.com/en-us/library/xz702w3e(VS.80).aspx

<%-- ... --%>

This is a server side comment, stuff you don't want anyone without code access to see:

    <asp:Label ID="lblAwesome" runat="server" />
    <%-- sometimes end users make me angry --%>
    <asp:LinkButton ID="lbEdit" Text="Edit" OnClick="Edit_Click" runat="server" />

http://msdn2.microsoft.com/en-us/library/4acf8afk.aspx

And that's that.