Download Method of controlling a device to perform an activity
Transcript
US007668990B2 (12) United States Patent (10) Patent N0.: (45) Date of Patent: Krzyzanowski et a]. (54) METHOD OF CONTROLLINGA DEVICE TO PERFORM AN AC TIVITY-BASED OR AN EXPERIENCE-BASED OPERATION (56) References Cited 5,414,426 A 5/l995 O’Donnell et a1. 5,519,878 A 5/l996 Dolin, Jr. Justin Flores, Boca Raton, FL (US); Glenn Harter, Wellington, FL (US) (Continued) (73) Assignee: OpenPeak Inc., Boca Raton, FL (US) (*) OTHER PUBLICATIONS Subject to any disclaimer, the term of this patent is extended or adjusted under 35 Total Remote Software.‘ Grg'?n Technology Downloads, 1 page, printed from InternetiURLihttp://WWW.grif?ntechn0l0gy.c0m/ software/softwareitotalremote.html, (printed Feb. 23, 2004). U.S.C. l54(b) by 477 days. (Continued) (21) Appl. N0.: 11/042,17s (22) Filed: Primary ExamineriNiketa l Patel (74) Attorney, Agent, or FirmiSterne, Kessler, Goldstein & Jan. 26, 2005 (65) Fox P.L.L.C. Prior Publication Data US 2005/0128048 A1 Feb. 23, 2010 U.S. PATENT DOCUMENTS (75) Inventors: Paul Krzyzanowski, FanWood, NJ (US); Notice: US 7,668,990 B2 (57) Jun. 16, 2005 ABSTRACT Related US. Application Data Activity-based or experienced-based operations are created (63) Continuation-in-part of application No. l0/783,5ll, and de?ned during an initialization or setup session for one or more system component, With minimum or no user interac ?led on Feb. 23, 2004, and a continuation-in-part of tion. The system components include legacy and/or non legacy devices, Which are con?gured and managed Within a application No. l0/782,923, ?led on Feb. 23, 2004, noW Pat. No. 7,024,256, and a continuation-in-part of controlled environment, such as a home or business peer-to peer or client-server network. During an initialization or application No. 10/783,017, ?led on Feb. 23, 2004, noW Pat. No. 7,129,855, and a continuation-in-part of application No. l0/387,590, ?led on Mar. 14, 2003. setup session, relevant device information and device meta data are collected via a conventional or discovery protocol (51) interface. For a legacy device that is not automatically dis coverable, a user adds the legacy device to the controlled Int. Cl. G06F 13/12 G06F 3/00 G06F 15/177 (2006.01) (2006.01) (2006.01) environment by inputting the device information. The device (52) US. Cl. ............................. .. 710/72; 710/8; 710/62; information can also be automatically collected from an accessible database. Once the device information has been 709/220; 709/221; 709/222 collected, the system components are con?gured to support (58) Field of Classi?cation Search ................. .. one or more activity-based or experienced-based operations. 710/72, 710/8, 62; 709/22(L222 See application ?le for complete search history. 27 Claims, 12 Drawing Sheets 100 ,/ 106(a) 104(b) ‘ 106(b) (Eg.. Audio/Video Stream 118 116 Leygucyrevlrces d ' ’ 130(a) Thermoslu? 114 130(b) 140 US 7,668,990 B2 Page 2 US. PATENT DOCUMENTS 5,537,463 A 5,552,917 A 5570085 A 7/1996 Escobosa et al. 9/ 1996 Darbee e131 10/ 1996 Bertsch 2001/0053274 2001/0054060 2001/0055954 2002/0006788 2002/0019984 5,689,353 A 11/ 1997 Darbee e131 A1 A1 12/2001 Roelofs et al. 12/2001 Fillebrown et al. A1 A1 A1 12/2001 Cheng 1/2002 Knutsson etal. 2/2002 Rakib 2002/0022991 A1 * 2/2002 5,771,388 A 6/1998 Mondrik et al. 2002/0031120 A1 3/2002 Rakib 5,802,467 5,805,812 5,875,108 5,898,386 5,926,108 A A A A A 9/1998 9/1998 2/1999 4/1999 7/ 1999 2002/0033760 2002/0034292 2002/0035403 2002/0035404 2002/0035605 A1 A1 A1 A1 A1 3/2002 3/2002 3/2002 3/2002 3/2002 Kobayashi Tuoriniemietal. Clark etal. F1000 etal. McDowell et al. 5930699 5,953,144 5,956,025 5956487 5959751 5963624 6,005,861 6026150 6,032,202 6,049,711 6,052,750 A A A A A A A 7/ 1999 9/1999 9/1999 9/1999 9/1999 10/ 1999 12/ 1999 2002/0035621 2002/0036795 2002/0068984 2002/0087746 2003/0103088 2003/0105854 2003/0163626 A1 A1 A1 A1 A1 A1 A1 3/2002 3/2002 6/2002 7/2002 6/2003 6/2003 8/2003 Zintel et al. Fujitani Alexander et al. Ludtke etal. Dresti et al. Thorsteinsson etal. Chen et al. A A 2/2000 2/2000 4/2000 4/2000 2003/0191826 2003/0204582 2003/0207668 2003/0208595 A1 A1 A1 A1 10/2003 10/2003 11/2003 11/2003 A A 6085236 A 6,131,028 6148205 6154745 6,167,046 6,170,007 A A A A B1 5812121118181 Fish @131 Ho?berg e1a1~ Kalhatsu Wicks er a113113113 Darbee er 31Goulden et 31Venkatraman e1a1~ Darbee er 31Pope Humpleman Fran1<e1a1~ Lea @1211 Ben-Yehezkel et 31Lea 7/2000 Lea__ 111/2000 11/2000 11/2000 12/2000 1/2001 Wh?mgton Cotton Kari er 31Terada e131Venkatraman e1a1~ Sharood et al. ............. .. 705/14 Bellingeretal. Shimoda etal. McFarland et al. Gouge et al. 2003/0220988 A1 11/2003 Hymel 2003/0224735 2003/0224807 2003/0236890 2004/0003051 2004/0003073 12/2003 12/2003 12/2003 1/2004 1/2004 A1 A1 A1 A1 A1 Moursund et al. Sinha et al. HurWitZ et al. KrZyZanoWski et al. Krzyznowski etal. 6,198,479 B1 3/2001 Hqmpleman e131 2004/0046677 A1 3/2004 Drestietal. 6,199,066 B1 6,199,136 B1 3/2001 Glltho e131 3/2001 Shteyn 2004/0047347 A1 2004/0054789 A1 3/2004 Worry etal. 3/2004 Breh etal. 6,208,341 6,208,866 6 a 218 a 931 6222 530 a a 3/2001 Van E9 9‘ 91' 2004/0071136 A1 4/2004 Laumen etal. 40001 AS g har et a1‘ 40001 Se queira 2004/0098456 A1 2005/0015764 A1 5/2004 Krzyzanowskl et al. 1/2005 Gaur 6,223,348 B1 40001 Hayes et a1‘ 2005/0108369 A1 6,243,707 6,243,772 6,259,707 6,266,612 6,275,865 6,282,714 6/2001 6/ 2001 7/2001 7/2001 8/2001 8/2001 B1 B1 B1 B1 B1 B1 B1 B1 B1 B1 6,288,716 B1 6,292, 554 6,317,028 6,317,718 6,321,092 B1 B1 B1 B1 3/2001 Rouhollahzadeh et al. Humpleman et al. Ghori et al. Dara-Abrams et al. Dussell et 31‘ Zou Ghoriet a1‘ 9/2001 Humpleman etal. 9/2001 11/2001 11/2001 11/2001 Oden et al‘ Valiulis Fano Fitch et 31, . >1< 5/2005 Sather et al. .............. .. 709/220 OTHER PUBLICATIONS Total Remote v2.0 user guide, 29 pages, printed from InternetiURLihttp://WWW.grif?ntechnology.com/downloads/ pdf/manuals/TotaliRemoteiuserimanual.pdf, (printed Feb. 23, 2004) International Search Report for International Application No. PCT/ US03/20189, Issued on Oct. 29, 2003. “Global Caché GC-100 Network AdapteriControl & Automate Common Devices OverA Network” GC-100 NetworkAdapter, URL 6,349,352 B1 2/2002 Lea http://WWWglobalcache.com/products/gc-index.htrnl, Oct. 11, 2004, 6,359,636 B1 3/2002 Schindler et al. 2 pages. 6,363,434 B1 6,388,399 B1 6,405,261 B1 3/2002 Eytchison 5/2002 Eckelet 31, 6/2002 Gaucher 6,463,343 6,496,135 6,512,754 6,563,430 6,587,067 6,597,374 B1 B1 B2 B1 B2 B1 6,600,421 B2 6,642,852 B2 10/2002 12/2002 1/2003 5/2003 7/ 2003 7/2003 7/2003 ll/2003 Emens et al. Darbee Feder et al. Kemink et al. Darbee et al. Baker et 31. Freeman DreSti et a1. “CC-100 Key Features and Bene?ts” GC-100 Network Adapter, URL http://WWWglobalcache.com/products/gc-features.htrnl, Oct. 11, 2006, 2 pages. “Global Caché GC-100 Network AdapteriControl & Automate Common Devices Over A Network”, Product Data Sheet, 2 pages, 2004. “IPL TSFI244, Two Serial, Four Flex I/O, andFourIR PortIPLinkTM Ethernet Control Interface With IR Learner”, EXtron IPL T SF1244 IP LinkTM Products, URL http://WWW.eXtron.com/product/product. asp?id:iplts?244&version?5rint, Oct. 11, 2004, 6 pages. “Creston Introduces RMC2e “ll/lini” Control System ” Home Toys 6,647,426 6,728,784 6,741,853 6,748,343 6,792,469 6,914,551 2001/0000194 B2 B1 B1 B2 B1 B2 A1 ll/2003 4/2004 5/2004 6/2004 9/2004 7/2005 4/2001 Mohammed Mattaway Jiang et a1~ AleXaIldeT e1 a1~ Callahan et al. Vidal Sequeira New Release, URL http://WWW.hometoys.con1/releases/mar03/ crestoni02.htrnl, Oct. 14, 2004, 2 pages. “Troy Serial Server,” URL, http://WWW.troygroup.com/Connectiv ity/products/SerialDeviceConnectivity/SerialServers.asp, doWn loaded Jul, 9, 2004, 2 pages, “Troy Serial Servers,” URL, http://WWW.troygroup.com/Connectiv ity/products/Datasheets/ 2001/0018663 2001/0036192 2001/0041561 200 1/ 0049275 2001/0049846 A1 A1 A1 A1 A1 8/2001 11/2001 ll/200l 12/ 2001 12/2001 Dussell e1 31~ Chiles et al. Venflllett et a1. Pierry et al. GuZZi et al. Troy%20Serial%20Server%20data%20sheet.pdf, downloaded Jul. 9, 2004, 1 page, “EtherWindTM 802.11bTM Wireless Print Server”, URL http://WWW. troygroup.com/Wireless/documents/datasheets/ TROYU%20EtherWind.pdf, Nov. 19, 2003, 2 pages. US 7,668,990 B2 Page 3 “EtherWindTM IEEE 802.11b Wireless Print Server”, URL http:// “Harmony RemoteiFeatures”, URL http://wwwharmonyremote. www.troygroup.con1/wireless/products/wireless/etherwind.asp, com/productifeatureshtrn, downloaded from the World Wide Web Nov. 19, 2003, 3 pages. “Troy Wireless Introduces EtherWind-Plus 802.11b Wireless OEM Board-Level Product; New Product Enables Wireless and Ethernet Connectivity on OEM Devices”, URL http://www.troygroup.con1/ wireless/company/news/08l302.asp, Nov. 19, 2003, 2 pages. “EtherWind-PlusTM 802.11b OEM Module”, URL http://www. on Feb. 9, 2004, 5 pages. troygroup.com/wireless/documents/datasheets/ TROY%20EtherWind%20Plus%20data%20sheet.pdf, Nov. 19, 2003, 2 pages. “EtherWind-PlusTM IEEE 802.11b OEM Connectivity Module”, URL http://www.troygroup.con1/wireless/products/wireless/ etherwindplus.asp, Nov. 19, 2003, 2 pages. “Speci?cations for Harmony Remote”, URL http://www. harmonyremote.com/productispeci?cations.htm, downloaded from the World Wide Web on Feb. 9, 2004, 2 pages. “HarmonyRemoteiFrequentb/Asked Questions”, URL http://www. harmonyremotecom/faqhtrn, downloaded from the World Wide Web on Feb. 9, 2004, 2 pages. “Harmony Remote SST-659”, published at http://www. harmonyremotecom, downloaded from the World Wide Web on Feb. 9,2004, 1 page. “Harmony Remote SST-748”, published at http://www. harmonyremotecom, downloaded from the World Wide Web on Feb. 9,2004, 1 page. “Pronto RF Extender, RFX 6000, Pronto accessory for operating “Harmony devices inside a closed cabinet, closet or even in another room”, URL harmonyremotecom, downloaded from the World Wide Web on Feb. 9,2004, 1 page. http://wwwpronto.philips.com/index.cfm?id:577f, Nov. 19,2003, 2 Remote SS T- 768t”, published at http://www. “Pronto RF Extender, RFX 6000, RF extender for TSU6000 & “Harmony Remote SST-659 User Manual”, dated Oct. 16, 2003, published at http://www.harmonyremote.com, downloaded from the TSU3000”, URL http://wwwremotecontrol.philips.con1/library/ World Wide Web on Feb. 9, 2004, 20 pages. documents/SPECiRFX6000.pdf, Nov. 19, 2003, 2 pages. “Harmony Remote SST-768 User Manual”, dated Mar. 25, 2003, published at http://www.harmonyremote.com, downloaded from the pages. “Harmony RemoteiOverview”, URL http://wwwharmonyremote. com/productioverviewhtrn, downloaded from the World Wide Web World Wide Web on Feb. 9, 2004, 37 pages. on Feb. 9, 2004, 2 pages. * cited by examiner US. Patent Feb. 23, 2010 Sheet 1 0f 12 US 7,668,990 B2 6;W..." ., A302 EomEw., 91N:Ao.vomw .{E tumoctwrc. 362 US. Patent Feb. 23, 2010 Sheet 3 0f 12 US 7,668,990 B2 300 ,1 303 Add or Remove Rooms 1 306 Assign WiFi/IR / Controllers to Rooms l 309 / Assign Detected UPnP/ Networked Devices to Rooms l 312 Add Legacy Devices / to Rooms i Con?gure Devices , 395 FIG. 3 US. Patent Feb. 23, 2010 Sheet 4 0f 12 wow fl2:,1.LI W$Q5E0:P .5 . US 7,668,990 B2 US. Patent Feb. 23, 2010 Sheet 5 0f 12 US 7,668,990 B2 wow .UEaw wow i@5E20:5 . wow wow US. Patent Feb. 23, 2010 Sheet 8 0f 12 US 7,668,990 B2 ooh F03 %; @gwmcEo:. *025;0E2aEL?03%0%252 9Mmw385E2:oN08I5»o?w .OEh US. Patent cow Feb. 23, 2010 Sheet 9 0f 12 US 7,668,990 B2 Now w.9“ k , .. Q52%9.0.2, ,mtmhmw 96W. 39:, 31 : 38$8Y- 3%$0 US. Patent Feb. 23, 2010 Sheet 12 0f 12 US 7,668,990 B2 /\/ EHO 1100 Processor 1 104 Main Memory 1108 Display Interface 1102 Display 1130 Secondary Memory 1110 Communication Infrastructu re Hard Disk Drive 1112 1 106 Removable Storage Removable Drive 1114 Storage Unit 1118 Interface 1120 Communications Interface 1124 Removable Storage Unit 1 122 X Communications Path 1126 FIG. 11 US 7,668,990 B2 1 2 METHOD OF CONTROLLINGA DEVICE TO PERFORM AN AC TIVITY-BASED OR AN EXPERIENCE-BASED OPERATION memory. HoWever, the user cannot inform the RCU of critical operational information of the controllable devices (e.g., tele vision, DVD player, etc.). Such critical operational information includes the input state (i.e., What signals are coming into What input). CROSS-REFERENCE TO RELATED APPLICATIONS For example, if a television is set to the tuner input (instead of the input for receiving DVD data) When a RCU is operated lian et al., entitled “User Interface for Multi-Device Control,” to play a DVD installed in a DVD player, the television Will display the broadcast channel to Which it is tuned at the time, and not the desired DVD. The user must physically select the incorporated herein by reference in its entirety. proper input by cycling through the available inputs using the This application is a continuation-in-part of US. patent application Ser. No. 10/783,511, ?led Feb. 23, 2004, by Kil input selection button. The lack of input state information also precludes the user from sWitching betWeen different activities This application is also a continuation-in-part of US. patent application Ser. No. 10/782,923, ?led Feb. 23, 2004, by KrZyZanoWski et al., entitled “Method, System, and Com puter Program Product for Automatically Managing Compo (e.g., Watching DVD, Watching broadcast TV, Watching cable) Without also physically changing from the current input state to the neW state, thereby placing the television in the proper input state. nents Within a Controlled Environment,” incorporated herein by reference in its entirety. Although the combination of improved computer process This application is also a continuation-in-part of US. patent application Ser. No. 10/783,017, ?led Feb. 23, 2004, by KrZyZanoWski et al., entitled “Device Control System, Method, and Apparatus,” incorporated herein by reference in its entirety. 20 devices connected to a residential netWork and facilitate the This application is also a continuation-in-part of US. patent application Ser. No. 10/387,590, ?led Mar. 14, 2003, by KrZyZanoWski et al., “Legacy Device Bridge for Residen tial or Non-Residential Networks,” incorporated herein by reference in its entirety. BACKGROUND OF THE INVENTION operations of these devices With minimal user intervention. 25 SUMMARY OF THE INVENTION A method, system, and computer program product are provided to con?gure and manage a plurality of system com 30 sumer electronics, and more speci?cally, to a process for 35 ling consumer electronics. 2. Related Art computer netWork can be time extensive and expensive to deploy. Additionally, there is no easy method to integrate, for example, home computer netWorks With other residential devices, such as televisions, stereos, DVD players, and other home electronics. Being able to ef?ciently distribute digital audio/video (AV) data among personal computers and other AV devices (such as, televisions, DVD players, PVRs, etc.) is ity-based or experienced based operations include favorite 40 trols, control macros (including context-based control mac ros), or the like. 45 dards and/or formats. 50 control netWorked residential devices. Although a remote control unit (RCU) can be trained to send signals to compo nents of an entertainment center (such as, a television, stereo, and DVD player), a conventional RCU cannot communicate and control multiple personal computers and other analog settings, playlists, security protocols, context-sensitive operations, user-speci?c operations, folloW-me system con complicated by differing and evolving communications stan Another signi?cant challenge is being able to effectively devices, and the controlled environment includes a peer-to peer communications system and/or a client-server netWork. In an embodiment of the present invention, “activity based” or “experienced-based” operations are created and de?ned during an initialiZation or setup session for a system component, With minimum or no user interaction. The activ The installation of computer netWorks (for homes and busi nesses) presents several challenges. For example, installing a ponents Within a controlled environment, such as a home, business, school, etc, as Well as its surrounding areas. The system components include legacy and/or non-legacy 1. Field of the Invention The present invention relates generally to control of con creating activity or experience-based operations for control ing capabilities and global access to information has resulted in signi?cant advancements in the information processing age, there exists a need for a simple, inexpensive, yet versatile system that can integrate the functions of multiple residential For non-legacy devices, relevant information about the system component is collected via a discovery protocol inter face. For legacy devices, a variety of system interfaces are provided for receiving the relevant information. In an embodiment, the system components (both legacy and non-legacy devices) are added, removed, or con?gured via a setup WiZard. During this initialiZation or setup process, relevant device information (e.g., device type, make, model, region assignment, etc.) is automatically discovered or added by user input. For a legacy device that is not automatically discoverable, the user adds the legacy device by identifying 55 and/or digital devices at a residence. In addition, a conven the device (by device type, make, model, etc.) and assigning tional RCU cannot communicate and control multiple resi the device to a particular room/region. The initialiZation or setup process is also designed to dential devices Without user intervention. Even With a con acquire “device metadata,” Which includes basic operational ventional RCU, a user must manually activate a hard key to send a command (in the form of infrared signals) to a con information or idiosyncrasies about each neW device (such as, 60 tuning rules, types and order of inputs, etc.). The concept of trollable consumer electronic (CE) device having direct line metadata is further described in the application entitled of-sight connectivity With the RCU. “Device Control System, Method, and Apparatus” (US. patent application Ser. No. 10/783,017). The present inven A conventional RCU suffers from other serious de?cien cies that limit its usefulness Without constant user interven tion. For instance in the “set up” process of a conventional RCU, the user can program the device by assigning a particu lar key With a series of key strokes that are stored in the RCU’ s tion includes mechanisms for optimiZing the collection of 65 device metadata With little or no human intervention. In an embodiment, a series of questions or informational prompts are dynamically optimiZed to avoid seeking irrelevant or US 7,668,990 B2 3 4 unnecessary information. In an embodiment, device metadata is contained in an accessible database. As such, the setup process can be tailored and streamlined based on the type and model number of the system component being added or con protocol interface, non-legacy devices are specially adapted for discovery and control by other devices.A “legacy device,” ?gured. The setup process is dynamically tailored to seek only that Which is speci?c to and necessary to control the discovery protocol interface. Lacking a discovery protocol interface, legacy devices, therefore, are not conventionally adapted for discovery and control by other devices. According to embodiments of the present invention during on the contrary, is de?ned herein to include any type of device or peripheral that is not capable of communicating via a particular system component in light of its user-speci?c con ?guration. Once the device information (including device metadata) has been collected, the system components (e.g., the assigned the initialiZation or setup phase, the system component is con?gured to perform one or more activity-based or experi legacy and non-legacy devices) are con?gured to support one enced-based operations With minimum or no user interaction. or more activity-based or experienced-based operations. For non-legacy devices, relevant information about the sys tem component is collected via a discovery protocol inter face. For legacy devices, a variety of system interfaces are provided for receiving the relevant information, as described in greater detail beloW. In an embodiment, the activity-based or experience-based BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES The accompanying draWings, Which are incorporated herein and form part of the speci?cation, illustrate the present invention and, together With the description, further serve to explain the principles of the invention and to enable one operations include one or more “favorite” settings for the the draWings, generally, like reference numbers indicate iden system component. For example, a favorite setting can be established for television programming, audio/video record ings, room temperature, hot tub controls, clock alarms, light/ dimmer settings, Web sites, neWs broadcasts, ?nancial chan tical or functionally or structurally similar elements. Addi nels, or the like. 20 skilled in the pertinent art(s) to make and use the invention. In tionally, generally, the leftmost digit(s) of a reference number identi?es the draWing in Which the reference number ?rst appears. FIG. 1 illustrates a residential controlled system according to an embodiment of the present invention. FIG. 2 illustrates a residential controlled system according to another embodiment of the present invention. In another embodiment, the activity-based or experience 25 30 FIG. 3 illustrates an operational ?oW for initialiZing system based operations include one or more “security” protocols for FIGS. 4a-4b illustrate a room control screen according to 35 can be prevented from accessing designated television chan nels, Web sites, areas (such as, lockable rooms, draWers, safes, 40 In another embodiment, the activity-based or experience nent can have one or more con?gurations based on a particu lar context. Depending on the type of system component, a 45 controls, or the like. Another con?guration can be established 50 coffee maker timer controls, or the like. A con?guration can timer controls, lighting, neWs broadcasts, or the like. The present invention describes methodologies and/or techniques for con?guring and managing a system compo 60 In another embodiment, the activity-based or experience based operations are “user-speci?c”. For example, a user speci?c con?guration can be established With security con trols set to block certain televisions programming, Web sites, audio recordings, or the like When in the company of minors. HoWever, When in the company of adults, a con?guration can be established With a loWer security setting. In another embodiment, the activity-based or experience based operations include “folloW-me” system controls. For instance, “folloW-me video” is implemented to transfer a implemented With a peer-to-peer communications system and/ or a client-server netWork. The system component includes legacy as Well as non-legacy devices. A “non sal Plug and Play (UPnPTM) Forum). Having a discovery for home of?ce activities, Which includes, Without limitation, presets for a baby monitor, a playlist of classical recordings, be established for morning rituals, Which includes, Without limitation, presets for alarm/snooze controls, coffee maker I. OvervieW ery protocol interface (e. g., interfaces de?ned by the Univer con?guration can be established for evening entertainment, Which includes, Without limitation, presets for lighting, audio/video presentations, security access Warnings, hot tub 55 legacy” device is de?ned herein to include any type of device or peripheral that is capable of communicating via a discov etc.), or the like. based operations are “context-sensitive”. The system compo DETAILED DESCRIPTION OF THE INVENTION nent of a controlled environment (such as, a residence, busi ness, school, or the like). The controlled environment can be the system component. For example, the system component can be con?gured to block certain content from being accessed by a designated user. Non-adult users, for instance, device metadata according to another embodiment of the present invention. FIG. 11 is an example computer system useful for imple menting the present invention. recordings can be prepared and/ or stored for playback on, for example, a television, monitor, or audio client. A sequence of graphic images or photographs can be prepared and/or stored for playback on, for example, a monitor or computer client. A playlist of other forms of media and/or multimedia can also be created according to embodiments of the present invention. In another embodiment, the activity-based or experience components in a controlled environment according to an embodiment of the present invention. an embodiment of the present invention. FIG. 5 illustrates a Wireless controller device control screen according to an embodiment of the present invention. FIG. 6 illustrates a non-legacy device control screen according to an embodiment of the present invention. FIG. 7 illustrates a legacy device control screen according to an embodiment of the present invention. FIG. 8 illustrates a legacy device control screen according to another embodiment of the present invention. FIG. 9 illustrates a device control screen for specifying device metadata according to an embodiment of the present invention. FIG. 10 illustrates a device control screen for specifying based operations include one or more “playlists” for the sys tem component. For example, a series of video or audio 65 selected video production to various displays throughout the controlled environment. As a user migrates from room to room Within the controlled environment, the user, or a system US 7,668,990 B2 5 6 component in communications With the user, is tracked to determine the user’s current location, and a display (e.g., monitor or television) positioned in the current location Will Similarly, the activity-based or experience-based operations herein by reference in its entirety; and in US. Pat. No. 6,792, 323 to KrZyZanoWski et al., Which is incorporated herein by reference in its entirety As described in greater detail beloW, activity-based or experienced-based operations are created and de?ned during can include “folloW-me audio,” “folloW-me lighting,” or the like. function and/or operation of system components, including automatically start to present a pre-selected video production. an initialiZation or setup session, and are executed With the In another embodiment, the activity-based or experience consumer electronic (CE) devices. The activity-based or based operations are de?ned by one or more control macros. experienced-based operations support the control of multiple A control macro includes a set of commands that, When devices of the same type (e. g., tWo DVD players). The activ executed, enables a controller to control multiple operations ity-based or experienced-based operations also support the and/or functions of one or more system components. In an segmentation of devices among multiple rooms (e.g., a tele embodiment, the set of commands are stored in a central location and are associated With a generic command. When vision in a living room versus a television in a bedroom); the control of non-IR, netWorked devices (such as devices having a discovery protocol interface); the control of devices beyond line-of-sight (via a Wireless netWork); and setup directly on a remote control device (as opposed to a ?xed stationary inter activated, the generic command retrieves and transmits the set of commands to the appropriate system component(s) for execution. In other Words, the present invention enables a remote or portable controller to transmit a single high-level face). request to a centraliZed command center (Which can be coupled to another computer). The centraliZed command cen 20 ter interprets the single request and ?nds the corresponding sequence of commands that needs to be transmitted to the II. Example Controlled Environment The present invention can be implemented in any type of controlled environment, such as a residence, business, school, appropriate system component(s). or the like. A residential controlled environment includes the In an embodiment, the control macro is de?ned to execute automatically upon the occurrence of a commencement parameter. The commencement parameter can be a pre con?nes of a home, apartment, mobile home, houseboat, or other types of residences. HoWever in embodiments, a resi dential environment includes the surrounding area of the resi dence, as Well as any shelters, constructs, improvements, or 25 de?ned time, date, state, event, or the like, or any combination thereof. For example, a pre-de?ned sequence of control com the like, Within a designated perimeter. In other embodiments, the present invention is imple mands can be executed upon the occurrence of a pre-set time (i.e., clock based) and/ or date (i.e., calendar based) automati 30 mented in a non-residential environment. A non-residential cally (i.e., Without direct user intervention). A user can desire to have the bedroom television automatically turned on and environment includes, but is not limited to, an of?ce complex, tuned to a particular channel at 6:30 am as part of his or her tainment arena, school or university, health care facility, suite of small of?ces, production studio, Warehouse, enter hotel, vacation resort, aircraft, ship, automobile, or the like. In “Wake up” routine. In another example, a pre-de?ned ticular system or component state. A control macro can be embodiments, the controlled environment for a non-residen tial embodiment includes not only the actual con?nes of the aforementioned structures but also their surroundings Within automatically executed When, for example, a television in the a designated perimeter. sequence of control commands can be executed upon the 35 occurrence of a particular event or the occurrence of a par Examples of a controlled environment are described in the family room has been poWered on. In an embodiment, the control macro is de?ned to be auto 40 application entitled “Method, System, and Computer Pro matically particulariZed for application in a speci?c context. gram Product for Managing Controlled Residential or Non Herein referred to as a “context-based control macro”, a con Residential Environments” (U .S. patent application Ser. No. trol macro can be particulariZed for use in any context (e.g., 10/382,897), and the application entitled “Method, System, and Computer Program Product for Managing Controlled any region, or any device of comparable type). In other Words, a speci?c system component or combination of components 45 Residential or Non-Residential Environments,” (U .S. patent application Ser. No. 10/180,500), both of Which are incorpo rated herein by reference in their entireties. As described in these applications, various methods and systems can be pro vided to manage the distribution of information (including 50 video, audio, voice, text, graphics, control messages, etc.) to is not identi?ed in the control macro. A context-based control macro is de?ned by specifying the “type” of system compo nent that is required to perform the activity-based or experi enced-based operation. Upon activation, the control macro is interpreted to select the appropriate system component(s). In an embodiment, the system component is selected by deter other devices and/or applications Within the controlled envi ronment. Such devices and/or applications include, but are not limited to, communications equipment (such as, tele mining the current location of the user (or a system compo nent in communications With the user). In another embodi ment, the user interacts With a remote orportable controller to specify the appropriate system component(s) at the time of 55 activating the context-based control macro. The above-mentioned activity-based or experience-based operations have been provided by Way of example and not limitation. Other types of activity-based or experienced based operations canbe used and are intended to be Within the phones, intercoms, etc.), entertainment systems (such as, televisions, CD/DVD players, gaming applications, stereos, etc.), monitoring systems (such as, security cameras, baby monitors, etc.), safety/security systems (such as, ?re alarms, sprinkler systems, locks on doors or WindoWs, etc.), personal 60 computers (such as, desktops, notebooks, notepads, personal digital assistants, etc.), cooking appliances (such as, ovens, scope of the present invention. Additional examples of activ coffee makers, electrical food/beverage Warmers, etc.), com ity-based or experience-based operations (including further fort systems (such as, heating and air conditioning (HVAC), descriptions of the above-mentioned operations) are humidi?ers, dehumidi?ers, air puri?ers, light sWitches, light described in the application entitled “Method, System, and Computer Program Product for Automatically Managing Components Within a Controlled Environment” (U .S. patent application Ser. No. 10/782,923), Which is incorporated 65 dimmers, etc.), poWer outlets, poWer supplies, or the like. An example of such controlled environments is shoWn in FIG. 1, Which illustrates a residential controlled system 100 according to an embodiment of the present invention. System US 7,668,990 B2 7 8 100 includes a communications network 180 that intercon Wireless local area netWork (LAN). Thus, communications netWork 180 includes Wired, Wireless, or both transmission nects a plurality of system components. The system compo nents include a positioning unit 102, tWo televisions 104 (shoWn as television 104(a) and 104(b)), tWo computer cli ents 106 (shoWn as computer client 106(a) and computer client 106(b)), a portable controller 108, a lighting device media, including satellite, terrestrial (e. g., ?ber optic, copper, UTP, STP, coaxial, hybrid ?ber-coaxial (HFC), or the like), radio, microWave, free-space optics, and/or any other form or method of transmission. As shoWn in FIG. 1, communications netWork 180 sup 110, a thermostat 112 for a HVAC system, a tuner 114, a ports peer-to-peer communications. HoWever, the present media player 116, a cable box 118, a digital satellite system (DSS) box 120, and a Wireless lighting control interface (such invention also supports client-server netWorks. FIG. 2 illus trates another embodiment of residential controlled system as a Lutron or X10 transceiver) 140. Other devices and/or applications can also be included as system components. 100 that includes at least one server 208. Server 208 polices Positioning unit 102 designates spatial locations Within the all tra?ic among the other system components. The exchange residence that serves as the hosting environment for system of information among the system components is routed or 100. Positioning unit 102 is coupled to the other system components (e.g., portable controller 108) via a Wired and/or Wireless interface. Positioning unit 102 is operable to desig otherWise controlled via server 208. As such, server 208 interacts With other system components to directly or indi rectly distribute data (including audio and/or video), voice, and/or control messages over communications netWork 180. In an embodiment, server 208 commands and controls the operations and/or functions of one or more of the other sys nate a ?oor or room Within the residence. Positioning unit 1 02 is also operable to designate a speci?c location or region Within a ?oor or room. Moreover, positioning unit 102 can be situated outside of the residence to thereby, designate external 20 ?le sharing, Internet access, and security. According to Computer client 106 includes a Wired and/or Wireless per sonal computer, personal digital assistant (PDA), enhanced telephone, personal television, or other data processing device linked to communications netWork 180. As a personal computer, computer client 106 can be a desktop, notebook, notepad, or the like. Portable controller 108 is a Wired and/or Wireless data processing device that enables a user to interact With, send control messages to, and/ or manage the distribution of infor tems components. The functions managed by server 208 include video serving, audio serving, telephony, messaging, areas of the residence. embodiments of the present invention, a user operates por table controller 108 to establish or recon?gure these functions 25 and/or receive media from server 208 or other systems com ponents or indirectly from the system components via server 208. In an embodiment, portable controller 108 includes several functions of server 208, as described herein, and manages the 30 mation (including audio, video, voice, and other data) among distribution of information (including audio, video, voice, and other data) among the other system components. In the other system components. Portable controller 108 can be another embodiment supporting peer-to-peer communica a portable version of the devices listed as computer client 106. For example, portable controller 108 can be a personal note tions, the system components exchange audio, video, voice, book or notepad computer, PDA, enhanced telephone, or other data, and/or control messages directly With each other 35 other device linked to communications netWork 180 and including a display With the ability to interact With the other system components. Hence, portable controller 108 enables a user to remotely control the operations of various components of system 100. In an embodiment, the display for portable controller 108 is capable of receiving video and/or audio from the other system components. In an embodiment, portable In an embodiment using a Wired transmission medium, communications netWork 180 is an Ethernet LAN having a 40 CAT-5 cable, or the like, distributed to each system compo nent, such as television 104, media player 116, etc. The sys tem component includes an audio/video (AV) connector that is responsive to receive the cable. In an embodiment, com munications netWork 180 includes a telephone line and/or controller 108 includes a ?ash ROM that enables Wireless doWnloads and/or uploads. Television 104 is a conventional television. In an embodi and Without being centrally managed. 45 poWerline (such as the communications technologies made available from the Home Phone Networking Alliance (Home PNA) or the like). ment, television 104 is enhanced to support interactive and/or personal services. Personal services include virtual record In an embodiment using a Wireless transmission medium, communications netWork 180 supports the IEEE standard ing, programming, pausing/reWinding live broadcasts, or the 802.11(a), Which speci?es a Wireless Ethernet protocol for large-siZed video. In another Wireless embodiment, commu nications netWork 180 supports the IEEE standard 802.1 1 (b), Which speci?es a Wireless Ethernet protocol for small-siZe video. In another Wireless embodiment, communications net Work 180 supports the IEEE standard 802.11(g). In another like. For example, television 104 can be a personal television enhanced to support the MSN® TV service, hosted by 50 WebTV NetWorks, Inc. (MountainVieW, Calif.), that supports the WEBTV® services available from Microsoft Corporation (Redmond, Wash). As shoWn, television 104 can be con nected to cable set-top box 118, DSS set-top box 120, and/or media player 116 (e. g., PVR, VCR, or DVD player). 55 Wireless interfaces With system 100. As shoWn in FIG. 1 and FIG. 2, communications netWork The aforementioned system components are not intended to be exhaustive. Other devices (including appliances), appli cations, and/or the like can be implemented, including, but 180 includes a Wireless netWork access point 182, such as not limited to, a refrigerator, stove, microWave, toaster, cof fee-maker, alarm clock, humidi?ers, sprinkler system, light 60 ing, light dimmers, etc. In an embodiment, portable controller 108 controls the operations and/or functions of such compo nents, such as on/off, timers, modulation (e.g., oven tempera tures, etc.), pause, snooZe, etc. As discussed, communications netWork 180 provides a transmission medium for communicating among the system components. Communications netWork 180 is a Wired and/or embodiment, the BLUETOOTHTM Wireless technology (de veloped by Bluetooth SIG, Inc.) is used to support short-range those available from Linksys, a division of Cisco Systems, Inc. (San Jose, Calif.), as a platform for the system compo nents. A Wireless access point 182 provides a central point for connectivity in a Wireless netWork and alWays-on connectiv ity necessary for tracking states of the system components. 65 Additionally, a Wireless access point 182 can provide a con nection point betWeen a Wired and Wireless netWork. In an embodiment using a server, as discussed above, a Wireless access point 182 serves as a platform for server 208. In alter US 7,668,990 B2 10 native server-based embodiments of the present invention, Co. Ltd. of Osaka, Japan, employs a proprietary protocol for server 208 can actually be located on a number of different discovery and control of that device in a network. device platforms in addition to a wireless access point (such In an embodiment of the present invention, one or more as, wireless access point 182), including a bridge device (such as, bridges 130(a)-130(e)), a personal digital assistant, a per bridges 130(a)-130(e) act as a plug and play proxy for a piece of legacy equipment and is herein referred to as a “plug-and sonal computer, or the like. play-enhanced” bridge 130(a)-130(e). The “plug-and-play enhanced” bridge 130(a)-130(e), therefore, includes a virtual System 100 also includes a plurality of infrared/ serial plug and play protocol interface that enables the plug-and play-enhanced bridge 130(a)-130(e) to advertise itself to bridges 130(a)-130(d), which comply with the IEEE 802.11 (b) standard for wireless communications. Each infrared/se other network entities as a conventional plug and play device, exhibiting attributes similar to the legacy device that is com rial bridge 130 interacts with one or more components. As shown, infrared/ serial bridge 130(a) interacts with television 104(a), cable box 118, and media player 116. Infrared/serial bridge 130(b) interacts with tuner 114 or any type of propri municatively connected to it. When plug and play commands arrive from the network (e.g., network 180), plug-and-play enhanced bridge 130(a)-130(e) translates them into an appro etary device that rely on infrared/ serial communications pro priate set of commands for controlling the legacy device, such tocols as would be apparent to one skilled in the relevant as IR commands or serial commands. Note that although this art(s). Infrared/ serial bridge 130(c) interacts with thermostat 112. Infrared/ serial bridge 130(d) interacts with lighting device 110. A wireless-Ethemet bridge 130(e) interacts with television 104(1)) and a DSS box 120. Wireless-Ethernet bridge 130(e) can support any IP addressable device. As such, television 104(1)) and DSS box 120 are “next generation” plug and play devices that have IP addresses. Therefore, the present invention can integrate legacy devices (e.g., CE devices that rely on infrared/ serial commu nications protocols), as well as devices having a discovery 20 oped, in part, by Apple Computer, Inc. of Cupertino, Calif. 25 embodiment, bridges 130(a)-130(e) are enhanced with a vir Forum) as system components. An example of a controlled environment implementing an IEEE 802.1 1(b) infrared/ serial tual plug and play protocol interface to enable “discovery and control” of legacy system components within system 100. It 30 Bridge for Residential or Non-Residential Networks” (U.S. patent application Ser. No. 10/387,590), which is incorpo 130(a)-130(e) can be modi?ed to include a virtual plug and play protocol interface that enables a legacy device, such as a conventional CE device, to be advertised to and controlled by non-legacy system components as if the legacy device were embodiments, bridges 130(a)-130(e) support other modes of wireless communications (e.g., RF, WiFi, BLUETOOTHTM, etc.) that enable discovery and/or control of legacy system 35 is modi?ed to enable discovery and/ or control of legacy sys with other system components of system 100. In other embodiments, a bridge 130(a)-130(e) can support a wired 40 interface to a legacy system component and a wireless inter face to another system components, a wireless interface to a legacy system component and a wired interface to other sys tem components, wireless interfaces to both the legacy sys tem component and other system components, and/ or wired 45 interfaces to both the legacy system component and other compliant with, for example, a packet-based discovery and Internet and Web protocols to enable devices such as personal system components. computers, peripherals, intelligent appliances, and wireless III. Setup Optimization devices to be connected to a controlled environment (such as, system 100) and to become automatically aware of each other. In accordance with a plug and play protocol (such as the UPnPTM Forum), when a user plugs a non-legacy device into a controlled environment (e.g., system 100), the device con ?gures itself, acquires a TCP/IP address, and uses a discovery protocol based on the Hypertext Transfer Protocol (HTTP) to announce its presence to other devices on within the con components. In another embodiment, a bridge 130(a)-130(e) tem components over a wired interface (e. g., serial or parallel) control protocol, such as those de?ned by the UPnPTM Forum. As would be appreciated by one skilled in the relevant art(s), the UPnPTM Forum is an open industry standard that uses should be noted that the wireless aspects of the present inven tion is not restricted to infrared communications. In other rated herein by reference in its entirety. In an embodiment of the present invention, a “virtual plug and play protocol interface” enables a legacy device to func tion similarly to a device having a conventional “plug and play” protocol interface. For example, one or more bridges Therefore in an embodiment, bridges 130(a)-130(e) are provided to “control” legacy system components. In another protocol interface (e.g., devices de?ned by the UPNPTM bridge is described in the application entitled “Legacy Device embodiment is described with respect to plug and play virtu alization, the present invention is not limited to plug and play virtualization, but also encompasses legacy device virtualiza tion in accordance with other packet-based discovery and control protocols, including but not limited to JiniTM, devel oped by Sun Microsystems of Santa Clara, Calif., or the RendezvousTM protocol, which is an open protocol devel 50 In an embodiment of the present invention, activity-based or experienced-based operations are created and de?ned dur ing an initialization or setup session with minimum or no user interaction, and are executed with the function and/ or opera 55 tion of various system components. This can be explained with reference to FIG. 3, where ?owchart 300 represents the general operational ?ow of an embodiment of the present trolled environment (e. g., system 100). Other plug-and-play invention for initializing or setting-up the system components compliant network entities can then negotiate a communica in a controlled environment, such as system 100. More spe ci?cally, ?owchart 300 shows an example of a control ?ow for tions protocol with the device, determine its capabilities, and control it. Note that although this embodiment is described with respect to the UPnP protocol, the present invention is not limited to the UPnP protocol, but also encompasses other standards-based discovery protocols, such as the Rendez establishing activity-based or experienced-based operations 60 ?ow is being described with reference to a “residential” con trolled environment, it should be understood that ?owchart 300 can also be expanded to support non-residential environ vousTM protocol, developed, in part, by Apple Computer, Inc. ments. of Cupertino, Calif. The present invention also encompasses proprietary discovery protocols designed for speci?c network devices. For example, the Panasonic® KX-HCM250 Wire less Network Camera, sold by Matsushita Electric Industrial during the initialization or setup phase. Although the control 65 The control ?ow of ?owchart 300 begins at step 301 and passes immediately to step 303. At step 303, a room or region within the controlled environment (e.g., system 100) is des