Download Computer user interface architecture that saves a user`s non

US007496830B2
(12) Ulllted States Patent
(10) Patent N0.:
Rubin et a].
(54)
US 7,496,830 B2
(45) Date of Patent:
COMPUTER USER INTERFACE
ARCHITECTURE THAT SAVES A USER’S
NON-LINEAR NAVIGATION HISTORY AND
5,237,648 A
ggrEgl?gGENTLY MAINTAINS THAT
Feb. 24, 2009
8/1993 Mills et al.
(Continued)
FOREIGN PATENT DOCUMENTS
(75) Inventors: Darryl E. Rubin, Redmond, WA (US);
Andrew C. Baird, Kirkland, WA (US);
EP
0342838 A
11/1989
John L. Beezer, Redmond, WA (US);
Jonathan C. Cluts, Redmond, WA (US);
_
Susan D. Woolf, Seattle, WA (US)
(comlnued)
(73) Assignee: Microsoft Corporation, Redmond, WA
(Us)
_
(*)
_
Notice:
OTHER PUBLICATIONS
Hirotsu et al. “CmeW/UiA Multimedia Web Annotation Sharing
_
_
_
Subject to any disclaimer, the term of this
System”, NTT Network Innovation LaboratoriesiJapan, IEEE,
1999, pp. 356-359.
patent is extended or adjusted under 35
_
U.S.C. 154(1)) by 420 days.
(Commued)
_
Primary ExamineriStephen Hong
(21) Appl' NO" 10/875’174
-
_
(22) Filed.
Assistant ExamineriMattheW J Ludwig
(74) Attorney, Agent, or FirmiBirch, Stewart, Kolasch &
Jun. 25, 2004
(65)
Birch’ LLP
Prior Publication Data
US 2004/0233235 A1
(57)
Related U.S. Application Data
.
.
.
A user interface architecture Wherein user content and user
.
interface are composed of documents With links is described.
(63) gonnémidggg of apghtc?ongg'oolg?saws’ ?led on
ec'
(51)
’
’ now
a '
’
’
'
User content documents and user interface documents are
both displayed in a single vieWing frame. A non-linear navi
gation history is maintained such that a user can navigate
G081? ‘17/00
(2006 01)
(52) U 5 Cl
along a ?rst path, back up using a previous link, navigate
715/206 715/205 715/208
' """"""""""" "
_
(58)
O‘
Int Cl
' '
ABSTRACT
Nov. 25, 2004
_
’ 7 1 5/24
_
71 5/853’
’
along a second path, back up along the second path using the
previous link, and re-navigate along the ?rst path again using
a next link. Every document page to Which a user navigates is
Field of Classi?cation Search ............ .. 715/500.l,
Saved in the user,s navigation history~ Users can query their
_ _
715/5011’ 206’ 205’ 298’ 240’ 853
See apphcanon ?le for Complete Search hlstory'
navigation histories and vieW their navigation histories in
various Ways. Users can access, annotate, and customize the
(56)
References Cited
user interface in the same manner in Which users access,
annotate, and modify user content documents.
U.S. PATENT DOCUMENTS
5,146,552 A
18 Claims, 11 Drawing Sheets
9/1992 Cassorla et al.
4011
A02
_
gunmnan link n m. mmmunm nu
mm- um um chm-y mum
dlspliy | m bum in m- um Wop-Hill
'
US 7,496,830 B2
Page 2
US. PATENT DOCUMENTS
5,239,466
RE34,476
5,347,295
5,390,138
5,434,929
5,434,965
5,471,568
5,559,942
5,572,643
5,623,679
5,625,833
5,630,125
5,632,022
5,644,674
5,666,113
5,680,636
5,682,439
5,687,331
5,689,717
5,717,860
5,717,879
5,719,595
5,727,129
5,729,687
5,737,599
5,745,116
5,760,773
5,761,485
5,761,683
5,784,058
5,786,814
5,790,818
5,801,685
5,801,687
5,802,516
5,819,301
5,821,925
5,822,720
5,826,025
5,832,263
5,835,092
5,838,313
5,838,914
5,845,262
5,854,630
5,860,074
5,862,395
5,877,757
5,893,126
5,893,132
5,918,236
5,920,694
5,923,326
5,924,104
5,931,912
5,933,139
5,933,140
5,937,416
5,940,080
5,948,040
5,950,214
5,956,034
5,956,048
5,978,818
5,982,370
5,983,248
5,986,665
6,011,537
6,012,055
6,018,334
6,018,342
8/1993
12/1993
9/1994
2/1995
7/1995
7/1995
11/1995
9/1996
11/1996
4/1997
4/1997
5/1997
5/1997
7/1997
9/1997
10/1997
10/1997
11/1997
11/1997
2/1998
2/1998
2/1998
3/1998
3/1998
4/1998
4/1998
6/1998
6/1998
6/1998
7/1998
7/1998
8/1998
9/1998
9/1998
9/1998
10/1998
10/1998
10/1998
10/1998
11/1998
11/1998
11/1998
11/1998
12/1998
12/1998
1/1999
1/1999
3/1999
4/1999
4/1999
6/1999
7/1999
7/1999
7/1999
8/1999
8/1999
8/1999
8/1999
8/1999
9/1999
9/1999
9/1999
9/1999
11/1999
11/1999
11/1999
11/1999
1/2000
1/2000
1/2000
1/2000
Morgan et al.
Norwood
Agulnick et al.
Milne et al.
Beernink et al.
Matheny et al.
Webb et al.
Gough et al.
Judson
Rivette et al.
Levine et al.
Zellweger
Warren et al.
Aihara et al.
Logan
Levine et al.
Beernink et al.
Volk et al.
Pritt
Graber et al.
Moran et al.
Hoddie et al.
Barrett et al.
Rothrock et al.
Rowe et al.
Pisutha-Arnold
Berman et al.
Munyan
Logan et al.
LaStrange et al.
Moran et al.
Martin
Miller et al.
Peterson et al.
Shwarts et al.
Rowe et al.
Carey et al.
Bookman et al.
Gramlich
Hansen et al.
Boudreau et al.
Hou et al.
Carleton et al.
NoZue et al.
Nielsen
Rowe et al.
Bier
Baldwin et al.
Drews et al.
Huffman et al.
Wical
Carleton et al.
Bittinger et al.
Earl
Wu et al.
Feigner et al.
Strahorn et al.
MenZel
Ruehle et al.
DeLorme et al.
Rivette et al.
Sachs et al.
Gaston
Lin
Kamper
DeRose et al.
Wrey et al.
SlotZnick
Campbell et al.
Eckerberg et al.
Bristor
6,018,344
6,018,742
6,025,841
6,034,689
6,035,330
6,037,934
6,038,598
6,049,812
6,052,514
6,054,990
6,055,538
6,064,384
6,072,490
6,076,917
6,081,829
6,091,930
6,094,197
6,105,044
6,122,649
6,133,925
6,141,007
6,144,375
6,144,991
6,151,622
6,154,771
6,157,381
6,163,778
6,181,344
6,184,886
6,195,679
6,195,694
6,199,082
6,205,419
6,205,455
6,211,871
6,226,655
6,230,171
6,243,071
6,243,091
6,266,772
6,271,840
6,272,484
6,279,005
6,279,014
6,289,126
6,289,362
6,301,590
6,320,169
6,320,577
6,321,244
6,331,866
6,331,867
6,340,980
6,356,287
6,369,811
6,370,497
6,377,983
6,389,434
6,393,422
6,397,264
6,405,221
6,418,421
6,421,065
6,425,525
6,437,793
6,437,807
6,446,110
6,457,013
6,460,058
6,484,156
6,486,895
6,490,603
6,529,920
>D
1/2000
1/2000
2/2000
3/2000
3/2000
3/2000
3/2000
4/2000
4/2000
4/2000
4/2000
5/2000
6/2000
6/2000
6/2000
7/2000
7/2000
8/2000
9/2000
10/2000
10/2000
11/2000
11/2000
11/2000
11/2000
12/2000
12/2000
1/2001
2/2001
2/2001
2/2001
3/2001
3/2001
3/2001
4/2001
5/2001
5/2001
6/2001
6/2001
7/2001
8/2001
8/2001
8/2001
8/2001
9/2001
9/2001
10/2001
11/2001
11/2001
11/2001
12/2001
12/2001
1/2002
3/2002
4/2002
4/2002
4/2002
5/2002
5/2002
5/2002
6/2002
7/2002
7/2002
7/2002
8/2002
8/2002
9/2002
9/2002
10/2002
11/2002
11/2002
12/2002
3/2003
Harada et al.
Herbert, III
Finkelstein et al.
White et al.
AstiZ et al.
Himuel et al.
Danneels
Bertram et al.
Gill et al.
Tran
Kessenich et al.
Ho
Bates et al.
Wen
Sidana
Mortimer et al.
BuXton et al.
DeRose et al.
Kanerva et al.
Jaremko et al.
Lebling et al.
Jain et al.
England
Fraenkel et al.
Rangan et al.
Bates et al.
Fogg et al.
Tarpenning et al.
Bates et al.
Bauersfeld et al.
Chen et al.
Ferrel et al.
Fiedler
Umen et al.
Himmel et al.
Borman et al. ......... ..
Paci?ci et al.
Shwarts et al.
Berstis
Suzuki
Finseth et al.
Martin et al.
Zellweger
Schilit et al.
Ishisaka
Van Der Meer
Siow et al.
Clothier
Alexander
Liu et al.
Eisenberg
Eberhard et al.
Ho
Ruberry et al.
Graham et al.
Knowles
Cohen et al.
Rivette et al.
Wone
Stasnick et al.
Levine et al.
Hurtado et al.
Walden et al.
Swaminathan et al.
Kaasila
Berquist et al.
Lection et al.
Saxton et al.
Koppolu et al.
Gupta et al.
Robertson et al.
Keenan et al.
Arons et al.
715/501.1
US 7,496,830 B2
Page 3
6,535,294
6,539,370
6,549,220
6,549,878
6,560,621
6,571,211
6,571,295
6,573,907
6,585,776
B1
B1
B1
B1
B2
B1
B1
B1
B1
3/2003
3/2003
4/2003
4/2003
5/2003
5/2003
5/2003
6/2003
7/2003
Graefe, et al: “Designing the muse: A Digital Music Stand for the
Symphony Musician”, Proc. of Conf on Human Factors In Comput
Arledge, Jr. et al.
Chang et a1.
Hsu et al.
Lowry et a1.
Barile
Dwyer et al.
Sidana
Madrane
Bates et a1.
ing Systems, Apr. 13-18, 1996, pp. 436-441.
Crespo, Chang, Bier: Computer Networks and ISDN Systems
“Responsive interaction for a large Web application: the meteror
shower architecture in the WebWriter II Editor”, 1997, pp. 1508
1517.
SOFTBOOK® Press The Leader in Internet-Enabled Document Dis
tribution and Reading Systems; http://www.softbook.com/con
6,631,496 B1*
10/2003
Li etal. ................. .. 715/501.1
sumer/reader.asp.
6,636,238
6,647,534
6,662,310
6,687,878
10/2003
11/2003
12/2003
2/2004
Amir et al.
Graham
Lopez et al.
Eintracht et a1.
Rocket eBook Using the Rocket eBook; http://www.rocketbook.
B1
B1
B2
B1
6,697,997 B1
6,710,790 B1
2/2004 Fujimura
3/2004 Fagioli
6,714,214 B1
6,760,884 B1
3/2004 DeMello
7/2004 Vertelney et a1.
com/Products/Faq/using.htrnl.
Explore the Features of the Rocket eBook; wysiwyg://212http://
www.rocketbook.com/Products/Tour/index.htrnl.
“Adobe Acrobat 3 .0 Reader Online Guide”, Adobe Systems, Inc., pp.
1-1 10.
6,766,494 B1*
7/2004
6,772,139 B1
6,788,316 B1
8/2004 Smith
9/2004 Maetal.
6,871,318 B1
3/2005 Wynblatt etal.
6,904,450 B1*
6/2005 King et al. ................ .. 709/203
2001/0031128 A1
Price et al. ................ .. 715/203
1/2002
2002/0097261
2002/0099552
2002/0194260
2005/0060138
7/ 2002 Gottfurcht et al.
7/ 2002 Rubin et al.
A1
A1
A1
A1
www.ebxwg.org/oebps/oebps1.0/download/oeb1-oebps.htm.
Kristensen, A., “Formsheets and the XML Forms Language” (Mar.
10/2001 Manbeck
2002/0013834 A1*
Esakov et al. ............. .. 709/223
0 822 501
0 890 926
0 902 379
WO 87/01481
WO 96/20908
W0 97 22109
WO 98/06054
WO 98/09446
WO 99/49383
A1
A1
A2
A
1999).
Dublin Core Resource Types: Structurality Draft: Jul. 24, 1997, Feb.
27, 1998.
Dublin Core Metadata Element Set, Version 1.1: Reference Descrip
tion 1999.
12/2002 Headley et al.
3/2005 Wang et a1.
Dublin Core Metadata Initiative: User Guide Working Draft Jul. 13,
1998 1999.
Greenberg, Saul, “A Fisheye Text Editor for Relaxed-WYSIWIS
FOREIGN PATENT DOCUMENTS
EP
EP
EP
W0
W0
W0
W0
W0
W0
KunikaZu, T., Patent Abstracts of Japan, Publication No. 11327789
for Color Display and Electronic Blackboard System, published Nov.
30, 1999.
Open eBook Forum, Open eBookTM Publication Structure 1.0, http://
Groupware”, Chi 1996 Proceedings, pp. 1 to 5.
Marshall, C., “Annotation from paper books to the digital library,”
ACM International Conf. on Digital Libraries, 1997, pp. 131-140.
“Joke eBook,” Jul. 1998, PrimaSoft PC, Inc., Electronic Book Series
Version 1.0, pp. 1-5.
2/1998
1/1999
3/1999
3/1987
7/1996
6/1997
2/1998
3/1998
9/1999
Munyan, D., “Everybook, Inc: Developing the E-Book in Relation to
Publishing Standards,” Sep. 1998, http://www.futureprint.kent.edu/
aIticles/munyan01.htm, pp. 1-7.
C. Marshall, “Annotation from paper books to the digital library,”
ACM International Conf. on Digital Libraries, 1997, pp. 131-140.
D. Munyan, “Everybook, Inc: Developing the E-Book in Relation to
OTHER PUBLICATIONS
Publishing Standards.” Sep. 1998, http://www.futureprint.kent.edu/
Kuo et al., “A Synchronization Scheme for Multimedia Annotation”,
aIticles/munyan0.1htm, pp. 1-7.
Multimedia Information Networking
PhotoImpact 3.0, Ulead Systems Inc., 1996.
Landay et al., “Making Sharing Pervasive: Ubiquitous Computing
Laboratory,
Tamkang
UniversityiTaiwan, IEEE, 1997, pp. 594-598.
“Huckle?ne De?ned”, Apr. 1990, “http://www.ylem.org/artists/
for Shared Note Taking,” IBM Systems Journal, vol. 38, No. 4, 1999,
mmosher/huck.html”, pp. 1-3.
“Tour of Korea Hypercard Stack”, 1996, Grose Educational Media,
“http://www.entrenet.com/~groedmed/kor2.html”, pp. 1-2.
BiZweb2000.comiscreenshots of an e-book, published on May
pp. 531-550.
1999,
pp.
1-4
“http://web.archive.org/web/*/http://www.
biZweb2000.com/eshots.htm”.
Landay, J. A., “Using Note-Taking Appliances for Student to Student
Collaboration,” 29th Annual Frontiers in Education Conference,
IEEE Computer Society, Nov. 1999, p. Session 12C4/15-12C4/20.
Adobe Acrobat 3.0 Review, downloaded from <http://www.hmung.
org/Reviews/>, reviewed Mar. 1997, pp. 1-18.
Schilit and Price: “Digital Library Information Appliances”, ACM
Conference on Digital Libraties, Jun. 23, 1992, pp. 217-226.
* cited by examiner
US. Patent
Feb. 24, 2009
Sheet 2 0f 11
US 7,496,830 B2
N0E
Now/
\mow
E219%;
US. Patent
Feb. 24, 2009
Sheet 3 0f 11
300
US 7,496,830 B2
302
getstate
defaulL/
Yes
i
display state —\ 306
308
hdetermine nature of user input
\determine region of user input
determine object associated with
region
determine action based on input
and object
perform action
FIG. 3
US. Patent
Feb. 24, 2009
Sheet 4 0f 11
400
begin
402
404
406
M)
it
get the next link in the current display state
1
i
examine the link's display pnoperties
display a frame based on the link‘s properties
display linked
to content?
Yes
i
examine link target
412
'
display link in frame
414
more
links in
the current
display
state‘?
FIG. 4
US 7,496,830 B2
U.S. Patent
Feb. 24, 2009
Sheet 6 0f 11
US 7,496,830 B2
rigggw __
W“mkzE6o.
.0E0
~22:5E2:.B“:8.60
US 7,496,830 B2
1
2
COMPUTER USER INTERFACE
ARCHITECTURE THAT SAVES A USER’S
NON-LINEAR NAVIGATION HISTORY AND
INTELLIGENTLY MAINTAINS THAT
HISTORY
ferent groups of users may be of different levels of computer
skill and have need of different sets of features, and the UI can
be customized to better suit their needs.
Users of prior art UIs sometimes become extremely frus
trated When their Work is lost because their Work Was not
properly saved. Accordingly, there is a need to provide a
This application is a continuation of and claims priority
from allowed application Ser. No. 09/456,975, ?led Dec. 7,
save-less model, so that users do not need to explicitly save
their Work.
Users of prior art UIs typically do not have a convenient
and seamless Way to record notes verbally and to associate
notes With particular parts of a document. Accordingly, there
is a need to provide rich support for audio note taking With the
1999.
FIELD OF THE INVENTION
This invention relates generally to computer user interface
architectures. More particularly, the invention provides a user
ability to correlate and synchronize audio and textual material
interface architecture in Which both user content and user
Prior art device-to-device and device-to-PC synchroniza
tion schemes typically are not seamless and require a great
deal of con?guration and attention from the user. Accord
ingly, there is a need to provide automatic and transparent
and to revieW and retrieve audio notes.
interface are composed of document pages With links.
BACKGROUND OF THE INVENTION
Many personal computer users ?nd the desktop metaphor
synchronization betWeen a user’s computers, such as a hand
20
of prior art computer user interfaces (“UIs”) confusing and
dif?cult to learn. Accordingly, there is a need for a system that
simpli?es the user’s interaction With the computer by using
feWer kinds of user interface controls in a more general Way.
Further, the Ways in Which users interact With information
about prior UIs is different than the Way the user interacts With
content, such as documents, presentations, and the like. For
example, in prior art UIs, content and UI information are
25
there is a need to make the process of getting help about a
function the same as the process for carrying out the function.
Prior art UIs typically have a “single-user model” at the
heart of their interface metaphors, Which makes sharing con
tent and annotations With other users dif?cult and non-intui
displayed entirely differently. Content is typically displayed
in a particular region or frame of the display. User interface
information is never displayed there. Instead, user interface
held computer and a desktop computer.
In prior art UIs, methods for getting help are currently
separate from the content and often require completely dif
ferent interactions than interacting With content. Accordingly,
30
tive. Accordingly, there is a need to make sharing and col
laborating on documents easier and more automatic.
information is displayed in dialog boxes, drop doWn menus,
SUMMARY OF THE INVENTION
and tool bars. User content never shoWs up in dialog boxes,
drop doWn menus, and tool bars. Similarly, users ?nd user
content documents and UI help information differently.
35
According to various preferred embodiments, the inven
tion includes a user interface architecture in Which user con
Accordingly, there is a need for a UI architecture in Which the
concepts and actions the user must learn are the same for
tent and user interface are composed of documents With links.
interacting With both content and the UI. Such a uni?cation
A link can relate a spot or region in a document With a spot or
makes computer softWare easier and more e?icient to use.
region in another document, so that touching the link causes
the display to navigate to that other document. A link can also
Prior art UIs for desktop computers typically require a
40
relate a spot or region in a document and an active runable
object such that When a user activates that link or touches that
keyboard and mouse in order for a user to interact With them,
and most pen-enabled palmtop computers have cumbersome
means of interaction. Therefore, there is a need for more
spot in the document, the associated object is run. Parameters
“natural” styles of interacting With a computer by using a
for the execution of the object may be supplied by properties
minimum number common gestures such as touch, hold,
45
Prior art UI desktop metaphors applied to small form factor
devices are typically cluttered and di?icult to use. In addition,
applications that provide rich functionality are sometimes
constrained by the limited ability of a user to navigate menus
50
and dialogs of prior art UIs. For instance, for such applica
tions, the menus and tool bars may get too big, and the help
system may get too cumbersome to navigate or search.
Accordingly, in addition to the need for a simpler more uni
?ed experience for the user of an application, there is also a
need to facilitate the uncluttered presentation of user inter
55
The path a user takes to reach a document typically affects
Usability data for prior art UIs shoW that users of multi
60
the behavior and presentation of the document. State-like
information for displaying a linkcd-to document page is
stored separately from the linked-from and linked-to docu
65
ments as part of the link’s properties. Users access, interact
With, and navigate among both user content documents and
user interface documents in a uni?ed Way, namely, by acti
vating links. Further, both user content document pages and
user interface document pages are displayed in a single vieW
WindoWed systems don’t alWays knoW Which actions Will
reduce the complexity and confusion sometimes caused by
multi-WindoWed user interfaces.
Prior art UIs typically offer limited capabilities for custom
izing the UI. Accordingly, there is a need for a UI architecture
that provides greater ?exibility to users, content developers,
and third-party softWare developers by providing broader
capabilities for easily customizing the UI. For example, dif
properties. Links can look like not only clickable spots, but
also ?ll-in ?elds and other kinds of Well-knoWn and later
developed user interface elements. Alternatively, links can
manifest in the containing document a frame displaying the
contents of part, or all, of the linked-to document. When links
are displayed, in addition to basing the display format of the
link on the link’s display properties, the link display format
may depend upon the characteristics of the linked-to docu
ment.
faces for applications providing very rich functionality.
produce results in Which WindoW. Therefore, there is a need to
associated With the link. Links, therefore, can act as com
mands. Links may be to any kind of command code.
A link may manifest in various Ways based on the link’s
erase, draW or Write.
ing frame. This uni?ed approach simpli?es the user’s inter
action With both user content and user interface documents by
US 7,496,830 B2
4
3
FIG. 8 depicts example Quick Help pages, one of Which is
reducing the number of concepts a user must learn in order to
access, interact With, and modify both user content docu
bookmarked, displayed on a personal vieWer according to a
ments and the user interface.
preferred embodiment of the invention.
A non-linear navigation history is maintained such that a
user can navigate along a ?rst path, back up using a previous
link an appropriate number of times, navigate along a second
FIG. 9 depicts an example of a pinned user content page
displayed on a personal vieWer according to a preferred
embodiment of the invention.
path, back up along the second path using the previous link an
appropriate number of times, and re-navigate along the ?rst
FIG. 10 depicts an example of inked annotation displayed
on a personal vieWer according to a preferred embodiment of
the invention.
FIG. 11 depicts an example End Page displayed on a per
sonal vieWer according to a preferred embodiment of the
invention.
path again using a next link. Every document page to Which a
user navigates is saved in the user’ s navigation history. Users
can query their navigation histories and vieW their navigation
history in various Ways, such as: by time, by appearance, by
site, document, section, page, and the like. User can also vieW
their navigation history as nodes With side tracking branches,
DETAILED DESCRIPTION OF THE INVENTION
as a linear list, or as a combination of most recently vieWed
pages and last feW task categories. According to a preferred
embodiment, navigation from user content pages through
Introduction
Clutter-Free and Simple
user interface pages that results in a command being executed
The UI architecture of this invention, also referred to herein
is automatically removed from the user’ s vieW of the naviga
tional history in order to keep the navigational history vieW
uncluttered.
A ?exible selection model is supported alloWing users to
select the object of a command either before or after the
command itself is selected. This ?exible selection model
alloWs UIs built according to the principles of this invention
to scale to small display areas. UIs built according to the
20
25
to page through a document and to folloW links, a user can
learn hoW to do any other UI operation. Signi?cantly, the
documents-With-links UI Works Without drop-doWn menus,
principles of this invention also scale Well to applications
having voluminous and/or complicated user interfaces by
facilitating an organiZed and uncluttered vieW of the user
interface command hierarchy and/ or user interface help infor
toolbars, WindoWs, or other cluttering UI elements (although
30
mation for such applications.
Users get at commands by navigating to a page Where the
desired command is found. In a preferred embodiment, the
documents-With-links UI according to the principles of this
invention is organiZed to make frequently used commands a
as a “documents-With-links UI,” supports creation of UI’s
that have essentially Zero clutter, and feW concepts to master.
In this Way, it is a major departure from prior art UI’ s using a
desktop metaphor. The user of a documents-With-links UI
according to this invention focuses primarily on content and
not on UI appurtenances. Starting With just knoWledge of hoW
some of these elements may optionally be made available
Where they are desired).
Content and UI are Uni?ed
In the UI architecture of this invention there is essentially
35
no distinction betWeen UI pages and content pages. “UI” and
“content” are the same thing, and exist in the same navigation
space. As described in more detail beloW, smart next/previous
single navigation step aWay, or through customiZations, no
steps aWay. A self-explanatory document, the Guide Book is
provided. The Guide Book is a readable manual that users can
logic and intelligent management of the navigation chain
solve technical problems caused by treating “UI” and content
go through in a logical order, a page at a time, like any
conventional user manual. Each command mention, hoWever,
is an active command instance that can be invoked in place.
as the same thing in a uni?ed navigational context.
Because there is no seam betWeen UI and content, no
notion of “dual” spaces, the documents-With-links UI is con
40
Other features and advantages of the invention Will become
ceptually simpler for the user than a model that has separate
apparent through the folloWing description, the ?gures, and
UI and content Webs. The one-space model is also more
poWerful and customiZable, as described in more detail
the appended claims.
45
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a schematic diagram of a conventional general
purpose digital computing environment that can be used to
implement various aspects of the invention.
so more experienced users can do the most common opera
50
FIG. 2 shoWs a conventional tablet and stylus-based com
puter that can be used to implement various aspects of the
invention.
FIG. 3 is a ?owchart shoWing simpli?ed steps at a high
level of abstraction for implementing a UI architecture
according to the principles of this invention upon start up.
FIG. 4 is a ?oW chart shoWing simpli?ed steps for imple
menting the “display state” step of the ?oW chart in FIG. 3.
FIG. 5 depicts an example Start Page displayed on a per
sonal vieWer according to a preferred embodiment of the
invention.
FIG. 6 depicts an example book cover page and table of
contents displayed on a personal vieWer according to a pre
ferred embodiment of the invention.
FIG. 7 depicts example Guide Book pages displayed on a
personal vieWer according to a preferred embodiment of the
invention.
beloW.
Shortcuts
Numerous UI shortcuts and direct manipulations may exist
as a con?gurable layer on top of the documents-With-links UI,
tions in context, Without navigating to UI pages. The user gets
the best of terse command access plus the richness of the full
broWser and ansWer system for exploring the command set.
The documents-With-links UI uses a Web architecture, With
UI shortcuts layered on top. A nai've user Will typically start
55
by using the documents-With-links UI Without the short
cutsithat is, by using the Guide Book to access UI functions.
An advanced user Will typically be able to perform all com
mon operations via the shortcuts and Without resorting to the
documents-With-links UI Guide Book as often.
60
Scalability to Various Display SiZes and Types
Various preferred embodiments of the documents-With
links UI Will be explained beloW in the context of a portable
“personal vieWer” platform. Nevertheless, the documents
With-links UI is scalable across a Wide range of device and
65
display types from desktop computers to laptops to hand-held
devices. Accordingly, the documents-With-links UI is
intended to be implemented on any type of computing plat
US 7,496,830 B2
5
6
form. The documents-With-links Ul exploits a large screen by
being able to show content in a book-like WayitWo full
side-by-side pages, as depicted, for instance, in FIG. 8. UI,
spot in a document and an active runable object such that
When a user activates that link or touches that spot in the
document, the associated object is run. Links, therefore, can
being content, takes advantage of the large format of pages
and the ability to turn/navigate pages (versus the smaller siZe
act as commands. Links may be to any kind of command
nonexistent provisions for navigation).
code. Scripts are one example. Binary code objects are
another example. As a result, pages that have links replace the
drop-doWn menus and dialog boxes of prior art Ul’s.
At the other end of the spectrum, the UI scales to small
screens because of the ?exible selection model that alloWs
command selection regions to be initiated either before or
after the desired command is chosen. This means, for
properties. Links can look like not only clickable spots, but
also ?ll-in ?elds and other kinds of Well-knoWn and later
developed user interface elements. A document page could
example, that on a small screen a user can call up a page of
have these other kinds of active elements that are really a form
command choices that completely obscure the original docu
of link. Accordingly, activating links goes beyond merely
ment due to screen siZe limitations, choose the command,
then return to the user document and then select the region for
clicking on colored underlined text and can include ?lling in
of menus/dialog boxes in prior art Uls and their limited or
A link may manifest in various Ways based on the link’s
?elds, pressing radio buttons, and the like. Again, pages With
links replace prior art dialog boxes having buttons and the
the command to operate on. Suppose a user Wanted to refor
mat some text, but had not selected the text to be reformatted
yet. The user could press a link that indicates that it Will
like. The links may look like buttons, but the links are simply
links.
Even the desktop metaphor of ?les and folders is expressed
reformat text. The UT of this invention Will display an indica
tor on the screen prompting the user to select the text to be 20 as lists of links on a page. Additionally, links have properties
that govern their appearance and behavior. For example, a
link’s properties may dictate that it appear as a blue under
reformatted. After selecting the text, the user can ?nish the
reformatting operation by clicking on another link, such as a
link that indicates that the selected text Will be reformatted.
Alternatively, the user could select the region ?rst and then go
?nd and invoke the command. Being able to select the object
upon Which a command Will operate after selecting the com
mand is unlike the selection model of prior art Uls Where
object selection must precede command selection. The selec
tion model of the documents-With-links Ul is discussed in
more detail beloW.
scored text string as in prior art broWsers, or as a 3D button, as
a graphic icon, as a thumbnail image of the content being
25 linked to, or even as an embedded frame that is open on the
content being linked to.
As described in more detail beloW, a user can modify hoW
30
content and the user interface.
General Concepts Underlying the Documents -With-Links Ul
Actions in a U1 according to the principles of this invention
Architecture
No Distinction BetWeen Content Documents and UI Docu
ments
35
The basic principle of the documents-With-links Ul is that
applies equally to both content and UI. These documents
could be implemented in HTML, XML, and the like. There is
the command parameters. Some commands may run With
canned parameters While others may present users With a
40
In fact the same document can mix content and UI. Even
though some Ul elements like context menus and toolbars
Because a command is a link, it has all the functionality of
links. For example, commands can visualiZe in multiple
Ways, such as an underlined blue label, a button, an icon, or a
45
tant for several reasons:
graphic image. They can be copied from one place to another,
moved, and deleted. Their properties can be manipulated,
such as to change their appearance or to preset some or all of
their parameters. Commands can also be created the same
Users only need to deal With one set of navigation controls
and conventions, and only one navigation space. Users
Way that any kind of link is created, via a Link command,
never need to think about Whether they are in content
space or Ul space.
Users can use the full poWer of the UI to manipulate the UI
itself. For example, users can search, annotate, custom
iZe and edit Ul pages the same as any content (subject to
50
permissions). Users can select from multiple vieWs for
the page being vieWed.
55
Which establishes a link betWeen a source and destination that
the user speci?es; in the case of creating a command link, the
destination is an executable object such as a command script
or binary. Everything users can do to a link, or to objects in
general (since a link is an object), users can do to commands.
The model naturally accommodates dynamic content,
including doWnloaded content that mixes content and UT
on the same page or as part of a netWork of related pages.
Links as Commands and Link Properties
form for entering the parameters (for example, the Properties
command).
may be presented in Ways that do not look document-like,
they are in fact implemented as documents and can be
manipulated as such.
The uniform treatment of content and “UI” pages is impor
occur by clicking on commands that exist on pages. To the
user, a command looks like a link, and in fact it is a link.
Speci?cally, a command is a link Whose source anchor is the
command hotspot, Whose destination anchor is the script or
code that implements the command, and Whose properties are
everything the user sees and interacts With is a document. This
essentially no difference betWeen content and UI documents.
a link manifests by modifying the link’s properties. The abil
ity to control a link’s appearance and behavioral properties
makes possible rich authoring and customization of both
60
A link can be an association betWeen tWo different spots in
a collection of document pages. The spots could be tWo spots
on the same page. The spots could be a spot on one page and
a spot on another page. Links can have properties that indicate
Guide Book
Users get at commands by navigating to a page Where the
desired command is found. The documents-With-links Ul is
organiZed to make frequently used commands a single navi
gation step aWay, or through customizations, no steps aWay.
Less commonly used commands may take more steps to get
to.
The documents-With-links Ul includes a self-explanatory
document, the Guide Book. This is literally a readable manual
relates tWo different places. What a link relates need not
that users can go through in a logical order, a page at a time,
like any current user manual. The difference is that each
command mention is an active command instance that can be
necessarily alWays be displayable pages. A link can relate a
invoked in place. A variety of Quick Help pages and indices
certain things about the link beyond simply the fact that it
65
US 7,496,830 B2
7
8
make it easy to get quick access to sets of commands that are
user’ s knoWn navigational patterns, such as Whether the user
commonly used together, that are logically related, or that are
typically used as part of a given scenario. Such Quick Help
got to the current node by navigating back by individual pages
or by groupings of pages (such as by site), or by linking from
pages could be assembled dynamically based upon the con
text of the document or documents being vieWed by the user.
a parent to a child. The Next function could include options to
present a user With a list of forWard choices, textually and/or
Users can also use search to ?nd commands. This could call
into play an intelligent user assistant or other conventional
this Way, the user can control Which branch to take, if the user
as a map in Which the user could Zoom into desired areas. In
help mechanisms When appropriate.
is not satis?ed With the documents-With-links UI’s selection.
Even if all commands Were only one navigational hop
aWay, hoWever, the documents-With-links UI Would not be
Part of What makes the navigation and context trimming
heuristics possible is built-in knoWledge of logical levels of
information grouping. For example, the documents-With
ideal, because users Want the most common commands to be
Zero hops aWay. Users also Want context sensitivity to com
mand presentation, like that provided by conventional context
links UI can include knoWledge of collections of pages mak
menus. The documents-With-links UI therefore accommo
ing up sections and chapters, of collections of sections and
chapters making up a document, of collections of documents
dates things like toolbars, context menus, and other UI short
making up Web sites, and so on. The same is true for the layers
cuts that the user can customiZe. Like everything else in the
documents-With-links UI, shortcuts are implemented as
documents With links. The UI shortcuts can be conceptual
iZed as being layered on top of the base documents-With-links
UI, yet shortcuts are actually constructed out of the same
of command ?nding and invocation. Such knoWledge of
semantic clustering helps guide decisions about popping con
texts and for presenting the user With reasonable choices
20
components that comprise the documents-With-links UI:
A problem With treating content and UI as part of the same
documents With links.
As a user looks up commands from the Guide Book, the
navigation space is that the user’s navigation chain gets clut
tered With UI-related pages. The documents-With-links UI
user Will learn short cuts so that the more a user interacts With
the UI, the less often the user Will typically need to navigate
to the Guide Book. Therefore, shortcuts Will be discussed
beloW, With reference to a preferred embodiment of this UI
architecture implemented on a personal vieWer, as the short
cuts Would appear to a user, because, eventually, shortcuts are
What most users Would use in their day-to-day activities With
the documents-With-links UI.
solves that through intelligent management of the navigation
25
30
Rich Navigation and Maintaining the User’ s Navigational
35
history map vieW. Ways in Which a user’ s navigational history
may be vieWed are discussed in more detail beloW in the
40
VieWing History section.
The history map vieW is a generaliZation of the linear
makes it easy for a user to revisit a place the user visited
previously, With important cues about the context in terms of
other places the user Was visiting at the time. The Next com
mand Works With the branching history too. A user can
explore a chain of links, back up, explore a different chain,
back up, and Next the user’s Way doWn the original chain to
45
get back to Where the user Was. This is much easier than
50
With-links UI knoWs What to act on, and What to remove from
the navigation context as folloWs. Commands operate on the
nodes that lie on the branch leading from the current selection
to the command. Further, additional heuristics may be used
Current selections in the documents-With-links UI are
similar to current selections in prior art desktop UIs, but there
are differences because the documents-With-links UI deals
With a netWork of active documents that are different than a
desktop of active WindoWs. The tWo schemes are similar in
that every document can have a single, possibly disjoint,
having to manually re-folloW the original chain of links,
of links they had previously folloWed
55
gation chain, the documents-With-links UI doesn’t forget all
selected area. The schemes differ in that the documents -With
links UI can’t use the idea of the current focus to decide What
selection a command should operate on. In prior art desktop
UIs, the document the user Wants to operate on typically has
the current focus, and all UI elements implicitly reference this
focus. In the documents-With-links UI World, the user may
have navigated several hops aWay from the document the user
60
Wants to operate on, as the user looked for the desired com
mand, so, in the documents-With-links UI, focus doesn’t dis
can use the history map and/ or the Next/ Previous commands
ambiguate anything.
to get back there again.
The Next function Works in this netWork- style navigational
context by using heuristics to pick Which path forWard the
user most likely intends. The most basic rule is to pick the
forWard path along Which the user backed to the current node.
Other rules provide additional intelligence to account for a
Book, resulting in a chain of several steps branching off from
B. When the user ?nally picks a command, the documents
for unusual cases.
Which Would be impossible if the user has forgotten the chain
the tWists and turns of Where the user has been just because
the user backs up and proceeds in a different direction. The
documents-With-links UI stores not only Where the user has
been, but also the path(s) the user took to get there. The user
text. Thus, after ?nding and executing the command, the
user’s navigation context is restored to document page B.
To ?nd a needed command the user might have to navigate
along a chain from D to several other pages in the Guide
current selection, and, in general, remove the navigation
history provided by prior art broWsers. The history map vieW
This is an important extension of the prior art broWsing
metaphor. Unlike prior art broWser UIs With their linear navi
context, and by making that context a true netWork, not just a
linear chain.
Speci?cally, When a user navigates from one place to
another, a neW branch in the navigation chain is started. So, if
a user’s context is currently B in the existing chain of docu
ment pages A-B-C, and the user navigates to D, then the neW
context is D. D might be a UI page the user navigated to from
document B. When the user clicks a command on page D, the
command executes and removes D from the navigation con
History
When a user performs a navigation action, a record is
created of Where the user navigated to and at What date and
time this occurred. It is thus possible to query this set of
records to derive many vieWs of a user’s navigational history,
including, for instance, a netWork vieW, also referred to as a
about points to jump to along the Next/ Previous chain.
Therefore, in accordance With a preferred embodiment,
65
instead of the current selection being the one Whose docu
ment has the focus, the current selection is the most recently
selected area. All commands that are con?gured to operate on
selections Will operate on that most recently selected area.
US 7,496,830 B2
10
Having executed, they Will trim the navigation context at the
a list should be a matter ofjust navigating to that list as a user
branch point that leads from the current selection to the com
mand itself.
Would in any other context and making the choice.
For the address book example, the documents-With-links
UI provides a link to the address book that is associated With
the input ?eld. The documents-With-links UI makes the link
from an input ?eldto its choice document, the address book in
this example, be a command With navigation behavior, as
The bene?t is that users are free to link into the Web of UI
pages, exploring them as necessary to ?nd the desired com
mand, and then to invoke it. The act of doing so Will end up
trimming all the UI navigation from the context, leaving the
user back Where the user Was before navigating to UI pages.
opposed to being an ordinary link. This command navigates
Note that depending on the length and content of the naviga
tional path betWeen the command and recent selection, the UI
the user to the document the user needs to choose from and
may shoW the user the target and prompt the user to con?rm
addresses the user Wants, the user can simply navigate back to
the send form Where the TO: ?eld is. Alternatively, the user
could close the address book or use an OK command, either
of Which Would return the user automatically. There is noth
ing to save, because all choice state is captured as the user
captures anything the user selects. After selecting the
before proceeding.
Note that if no selection exists When a command is
executed, then the next selection the user establishes Will be
considered to be the most recent selection for purposes of
determining the command target. In this case, after the user
proceeds. If there is a change in plan, the user could simply
makes the selection, command buttons Will be presented in
cancel the current selection, or clear the TO: ?eld When the
context With the selection by Which the user can con?rm or
user returns.
cancel execution of the previously selected command. Of
course, other suitable methods of determining What selection
20
to operate on are also possible. For instance, one such method
is to alloW only a single selection to be extant at a time. That
is, any time a neW selection is initiated, any prior selection is
cancelled (un-selected). Another method is to choose What
selection to operate on by doing a backWards scan in time
order through the navigation context until an active selection
is found. In most cases envisioned by the inventors, this latter
highlighted. The current choices are highlighted because the
command that takes the user to the address book picks up the
TO: state and paints the required selection regions.
25
To facilitate making selections in scenarios like this
address book example, an additional facility is provided. The
command Which presents the document to choose from can
also cause checkboxes to appear next to each entry in the
solution produces the same result as the ?rst one discussed;
i.e., the most recent selection Will be found.
The effect of the algorithm for trimming the navigational
If the user later Wants to change the TO: ?eld choices, the
user may click the TO: ?eld link again, and the user is taken
back to the address book, With all the current choices still
document. Rather than manually selecting items in the docu
30
ment, the user can check or uncheck the checkboxes. This
causes the associated entry to be selected or unselected,
context is typically to isolate and remove the branch Whose
purpose Was to ?nd the command that Was just executed.
respectively.
Trimming the navigational context of navigation to UI pages
does not alWays occur, hoWever. For example, anApply com
mand for property setting could leave the property form active
dard document, the address book, and the normal multiple
In this address book example, the user is accessing a stan
selection idiom to make and change the user’s To: ?eld ?ll-in
choices. The user has the full poWer of the normal UI for
and not trim the navigation context. Of course, different com
mands may choose to operate on the navigation context in
different Ways. Nevertheless, a couple standard Ways Will
navigating, vieWing, and searching, the address book and can
navigate to other documents containing addresses Where the
typically cover most of the cases.
Flexible Selection Model
The documents-With-links UI does not require users to
40
address book, the user Will see the TO: items highlighted in
the address book only if the user links to the address book (or
successor nodes) via the TO: ?eld in question. Thus, the
make a selection before invoking a command. Users are free
to select ?rst and then click a command, or click the command
?rst and then make a selection. If an appropriate selection
does not exist When a command is invoked, the user is
prompted to make a selection at that time. The selection
mechanism and command verbs are designed to give users a
45
lot of latitude about the order in Which they do things When
carrying out commands. Besides ?tting better to users’ per
sonal habits, this makes it harder for users to do something
“Wrong,” such as inadvertently applying formatting to text
50
This is a signi?cant principle of the documents-With-links
to achieve state-like behavior Without requiring special
modes or UI mechanisms like dialogs. The implementation of
chooser ?elds is one of the more important uses of this con
cept.
Modeless UI
55
The documents-With-links UI may display the same page
of a document differently depending upon Which link navi
Unlike prior art UI’s, the documents-With-links UI is
essentially modeless. For example, a user could be involved
in ?lling out a form for carrying out a UI operation such as
creating a formatting style for a table. In the middle of doing
this the user could navigate aWay from this UI form and get
engaged in some other UI operation, such as ?lling out
gated a user to the page. Suppose the user Wants to ?ll in the
TO: ?eld of an e-mail message. In this situation, the user
Wants to open the address book and make possibly several
existence of an active To: ?eld does not interfere With other
uses of the address book or With other active To: ?elds.
UI: the path a user takes to reach a document typically affects
the behavior and presentation of the document. This is a Way
other than the text the user Wants to reformat.
Page Display Depends Upon the Link Used to Navigate to
the Page
user can make other choices. Note that because the selection
state is associated With the path the user took to reach the
60
another form for a different purpose, and then, at any time,
choices. To accomplish this Within the document/navigation
metaphor, forms couldhave special-purpose chooser controls
navigate back to the original, incomplete form. There is no
restriction on the number of such incomplete operations that
Where this provides a good shortcut for the most common
choices. But the chooser UI should also make it possible to
on sWitching aWay from such incomplete operations. This is
leverage the full poWer of the navigation, broWsing, vieWing,
and search UI that is available in list-oriented documents like
?le folders and the address book. Making a choice from such
can be in progress simultaneously. Nor is there any limitation
65
unlike the prior art, Where complex operations, typically per
formed via dialogs, must generally be complete or cancelled
before the user sWitches to another activity. Unlike prior UI ’ s,
US 7,496,830 B2
11
12
the user of a documents-With-links UI is typically not
types of computer readable media Which can store data that is
accessible by a computer, such as magnetic cassettes, ?ash
restricted from interrupting one operation to initiate another,
memory cards, digital video disks, Bernoulli cartridges, ran
dom access memories (RAMs), read only memories (ROMs),
and the like, may also be used in the example operating
or several others, nor Would such an interruption cause the
user to lose Work already performed in partially completing
the ?rst operation.
Creating and Saving Information
environment.
Another architectural topic central to the documents -With
A number of program modules can be stored on the hard
links UI is the model for creating and saving information. Any
disk, magnetic disk 190, optical disk 192, ROM 140 or RAM
time a user creates something neW, the user is creating a neW
150, including an operating system 195, one or more appli
cation programs 196, other program modules 197, and pro
document and linking it into a context. For an object inserted
into an existing document, such as an embedded annotation,
the user is linking it into the document that is to contain it,
gram data 198. A user can enter commands and information
into the computer 100 through input devices such as a key
in place (OLE-style embedding). Physically, the object is
board 101 and pointing device 102. Other input devices (not
shoWn) may include a microphone, joystick, game pad, sat
stored as a child Within the parent document’s container.
For neW, standalone objects, like a neW Word processing
ellite dish, scanner, or the like. These and other input devices
are often connected to the processing unit 110 through a serial
document, the object is instead added to the current naviga
port interface 106 that is coupled to the system bus, but may
be connected by other interfaces, such as a parallel port, game
port or a universal serial bus (U SB). A monitor 107 or other
type of display device is also connected to the system bus 130
With link properties implicitly set to make the object visualiZe
tion context, as if the user had done a Next to it from Wherever
the user Was When the user issued the NeW command. Physi
cally, the object may be stored in the user’s sea of “free space”
20
in a hidden system folder, not part of any folder the user is
via an interface, such as a video adapter 108. In addition to the
aWare of, unless and until the user chooses to ?le it some
monitor, personal computers typically include other periph
Where.
The user does not need to put documents into a ?ling
hierarchy or save them. This is because the documents-With
links UI stores all navigational history. Accordingly, users can
eral output devices (not shoWn), such as speakers and print
ers.
25
?nd the documents they create by vieWing or searching their
The computer 100 can operate in a netWorked environment
using logical connections to one or more remote computers,
such as a remote computer 109. Remote computer 109 can be
history map. A user could ?le a document into a folder as an
a server, a router, a netWork PC, a peer device or other com
optional step, using a Save As command or by manually
creating a link in the folder that leads to the document (the
Save As command could be simply a shortcut for creating
mon netWork node, and typically includes many or all of the
30
elements described above relative to computer 100, although
only a memory storage device 111 has been illustrated in FIG.
such a link). Also, the user could use Save/ Save As to update
1. The logical connections depicted in FIG. 1 include a local
or create versions of a document in the ?ling hierarchy as
area netWork (LAN) 112 and a Wide area netWork (WAN)
113. Such netWorking environments are commonplace in
desired.
35
EXAMPLE HARDWARE PLATFORMS FOR
IMPLEMENTING ASPECTS OF THE
INVENTION
FIG. 1 is a schematic diagram of a conventional general
purpose digital computing environment that can be used to
Internet.
When used in a LAN netWorking environment, the com
puter 100 is connected to the local netWork 112 through a
netWork interface or adapter 114. When used in a WAN net
40
includes a processing unit 110, a system memory 120, and a
45
system bus 130 may be any of several types of bus structures
including a memory bus or memory controller, a peripheral
bus, and a local bus using any of a variety of bus architectures.
50
A basic input/output system 160 (BIOS), containing the
basic routines that help to transfer information betWeen ele
ments Within the computer 100, such as during start-up, is
stored in ROM 140. Computer 100 also includes a hard disk
drive 170 for reading from and Writing to a hard disk (not
shoWn), a magnetic disk drive 180 for reading from or Writing
to a removable magnetic disk 190, and an optical disk drive
191 for reading from or Writing to a removable optical disk
can be operated in a client-server con?guration to permit a
user to retrieve Web pages from a Web-based server. Any of
55
program modules and other data for the personal computer
100. It Will be appreciated by those skilled in the art that other
various conventional Web broWsers can be used to display and
manipulate data on Web pages.
FIG. 2 shoWs a tablet and stylus computer that can be used
in accordance With various aspects of the present invention.
Any or all of the features, subsystems, and functions in the
60
are connected to the system bus 130 by a hard disk drive
interface 171, a magnetic disk drive interface 193, and an
optical disk drive interface 194, respectively. The drives and
their associated computer-readable media provide nonvola
tile storage of computer readable instructions, data structures,
may be stored in the remote memory storage device.
It Will be appreciated that the netWork connections shoWn
are example and other means of establishing a communica
tions link betWeen the computers can be used. The existence
of any of various Well-knoWn protocols such as TCP/IP, Eth
ernet, FTP, HTTP and the like is presumed, and the system
192 such as a CD ROM or other optical media. The hard disk
drive 170, magnetic disk drive 180, and optical disk drive 191
Internet. The modem 115, Which may be internal or external,
is connected to the system bus 130 via the serial port interface
106. In a netWorked environment, program modules depicted
relative to the personal computer 100, or portions thereof,
The system memory includes read only memory (ROM) 140
and random access memory (RAM) 150.
Working environment, the personal computer 100 typically
includes a modem 115 or other means for establishing a
communications over the Wide area netWork 113, such as the
implement various aspects of the invention. Computer 100
system bus 130 that couples various system components
including the system memory to the processing unit 110. The
o?ices, enterprise-Wide computer netWorks, intranets and the
65
system of FIG. 1 can be included in the computer of FIG. 2.
Computer 201 includes a large display surface 202 (e.g., a ?at
panel display) on Which a plurality of WindoWs 203 is dis
played. Using stylus 204, a user can select, highlight, and
Write on the display area. Computer 201 interprets marks
made using stylus 204 in order to manipulate data, enter text,
and execute conventional computer application tasks such as
spreadsheets, Word processing programs, and the like. One