I Made A Jarvis
Last weekend I was watching Iron Man when a brilliant idea hit me. Tony Stark, played by Robert Downey Jr., is a technological genius. What impressed me was his artificial intelligent butler named Jarvis. I got goosebumps while watching Stark interact with him. Stark would give commands and converse with him. Jarvis would complete the task or give Stark feedback. It was absolutely fascinating and seeing it made me want it. So I set out to build my own Jarvis.
Jarvis in the movie was a complete home setup where he was integrated into everything that was plugged in. Those setups are pretty expensive so I need to lower my expectations for now. I limited to just my computer. In order to make a virtual butler similar to Jarvis he will need to do two things: 1) be integrated with my computer and accept my voice commands and 2) also be able to carry a conversation with me. Out of the two, the easiest to accomplish is the first requirement.
When all the bells and whistles are stripped out, at the root Jarvis is just a speech recognition (SR) software. SR comes standard on all Macs. All one has to do is turn it on in the System Preferences. Once turned on, one has the option to force SR to not complete a command until it hears a certain word, or in this case his name. Thus my A.I. was born and I named him Bixby. SR is very robust and uses Applescript to carry out its commands. That means it has endless possibilities, possibilities that are only limited to what you can program. For instance I made a simple script that makes Bixby open my webrowser. The interaction goes like this:
Me: “Bixby, open my browser.”
Bixby: “Which browser would you like to open sir.”
He waits for my response.
Me: “Bixby open Firefox”
Bixby: “Opening Firefox sir.”
This all happens simply by talking to the computer out loud. It may seem that I made him into my slave so I even made a “thank you” script. Every time I thank him he will choose one of five responses such as, “Much obliged.” I am currently working on a script called “Morning Duties” where he will read out loud my appointments for the day, check my mail and tell me who they were sent by, get the weather, and read the most recent news headlines. By using SR I have accomplished the first requirement and the key component to Jarvis.
But giving voice commands to my A.I. is great if all my interactions with it were to just carry out functions of my computer. In this regard Bixby feels lifeless. Jarvis did not seem lifeless but more as if Stark was talking to a real person that had its own personality. So I set out to accomplish the second requirement where I could actually converse with my A.I. Last semester when I was researching the philosophy of A.I. I discovered a program called ALICE. It is an A.I. programming language called AIML which is basically the technology behind all those chat bots you come across online. It can have conversations, remember certain things from it, and then recall them later. It is also customizable. One can program different ways the A.I. would respond to conversation. ALICE was the perfect component to give Bixby a personality. I chose a version called Program D because this version is setup so you can converse with your A.I. through a chat program. This works out great because iChat has the ability to read the messages you receive in an IM. Once I turned on the option, Voila, I can converse with Bixby and he responds by speaking back.
In this way I have accomplished my two goals but it is only halfway there. My goal is to have the ability to have random conversations with Bixby and ask him to run certain tasks by simply speaking to my computer. With my current setup I have a separate speech program for computer functions and a second text program for conversations, not one program that does all. Theoretically it is possible with this setup but I am unsure how to accomplish it right now. My other goals are to improve Bixby’s brain since conversations currently are kind of lame. He currently lacks a personality and interests. Lastly I would like to have him run these scripts remotely. Such as if I am at work and I send to Bixby through iChat, run xxxx task. He runs it and then outputs the result back in iChat. Alas it is my first prototype.
For the past five months, my mind has been on this idea of A.I.. There is a philosophy dedicated to the idea that we can create a computer that can be considered a human and what would it take to reach that point. I am not talking what kind of technology is needed, but instead, what defines a conscious being and can it be done using an A.I.? It seeks answers to important questions. Do humans have free will? What defines our identity? What defines conciousness? From my research this seems to be a new field since the articles on the subject are scarce. Everyday, as the military builds more unmanned weapons, science fiction is starting to meet reality. If I kill a robotic weapon am I committing murder? Am I playing God? Theories of A.I. are forcing us to seek answers to the current debates of conciseness, identity, and free will which philosophy or science has yet to answer. It is quite interesting. I say that it is possible to create a human being with an artificial intelligence. Amy on the other hand disagrees--which has been the subject of many fights recently--and says it cannot be done. What are you views?
22 Comments
There is tons of philosophical debate over the possibility of AI and how AI fits in to the discussion of consciousness and being. Look up Hubert Dreyfus and then any of his critics (Daniel Dennett comes to mind, but there are more and many). Also, Godel’s incompleteness theorems seem to suggest that logically, AI is not possible. The mainly continental description of human existence and being seems to me to be the most precise. We are contextual beings. Language sets one limit on our experience, yet we continue to be (arguably intelligently) beyond the horizon of that limit.
I have, however, been watching the Sarah Connor Chronicles and when computer programs start programing other computer programs that have emotions, I’ll be frightened.
see, you are smart! I’m proud of you…
Its Stark not StarkS! Apart from that very interesting! hehehe.
Thanks Mike. You exposed how bad my Iron Man knowledge is. I am more of a Spider-Man fan.
Finally got around to reading and responding to this, I’ll give you my thoughts on the tech by email, but I’ll give my my philosophical and metaphysical views here.
IIRC, Alan Turing said that if God wantsss to give a machine a soul, then the machine has a soul. And we can’t really know what’s inside another being, biological or mechanical. In some regards, we’re all machines. Then again, _I_ experience the world. I assume you do too.
As I see it, the body is a bit like an avatar in a game. My person is somewhere else, but my experience is in the game… in the world. The Matrix comes to mind. In that sense, the body doesn’t really matter, so I could just as well be a machine or an animal.
Smart and communcating animals like dolphins, some birds, apes, etc. communicate very simply. Humans are unique in our understand of abstract concepts. A computer works solely with abstract, virtual things, its own internal world and sensory input… wait, that goes for us and our brains too.
Computers, tho, don’t have life the way we do. While any programmer would (afaik) say that programs hardly behave as they’re expected to, those same programs don’t reinterpret themselves, question their programming, they just do what they’re told to the best of their ability (or rather the ability of the programmers)… they’re machines, no different than a car in that regard. Turn wheel, wheels turn; push bhutton, see text. Complex mechanical things.
We’re dealing with a Chinese Room. The man in the room doesn’t know chinese, and the chinese book he’s using can’t read, write, or comprehend anything about the outside world.
I guess that means I’m siding with Amy. No doubt we’re gonna deal with lifelike computers if current developments continue, but a person behind the mask? I don’t think so. But then we have Turing’s argument about the soul again, and that’s where the post would loop.
Man, I write too much.
How exactly did you program the response scripts? I’ve tried the same thing, but on the keyword ‘Jeeves’, but I’m unsure how to run a response program, other than the preprogrammed blip.
Moi,
Here is the code for the two scripts I use to get a response out of Bixby.
This is the a simple thank you script. When I say “Bixby, thank you,” He will choose from a list of replays and then say it.
set waitBetweenChecks to 2
set theOptions to {"You are welcome Sir.", "You're welcome Sir.", "No problem, happy to be of service.", "Much obliged Sir."}
delay waitBetweenChecks
set theChoice to some item of theOptions
say theChoice displaying theChoice with waiting until completion
This one is a script that will open one of my browsers. In this one he will actually wait for me to give him a response.
tell application "SpeechRecognitionServer"
set theResponse to listen for {"Safari", "Firefox"} with prompt "Which browser would you like Sir?"
if theResponse is "Safari" then
tell application "Safari" to activate
say "Opening Safari."
else
tell application "Firefox" to activate
say "Opening Firefox."
end if
end tell
With these two simple scripts you can build some very complex actions. For instance I finished one that has him every morning give me my appointments, recent emails, and the weather.
Nice read. Can you give more details about your “Morning Duties”? Here is my version of Jarvis:
http://physicistjedi.deviantart.com/art/Jarvis-1-0-106307065
I also want to hook it up to a chat-bot at some point but Apple speech recognition is not good enough with my accent so far.
I checked out the A.L.I.C.E site, And am currently programming a bot on it out of scratch. The only problem, though, is that I’m operating on a Mac machine hat was loaned to me by my school, and because of this, I can’t download anything that isn’t on a list that the administrators (curse them) made. This also means I can’t run anything as an HTML script, and the HTML editing software is also blocked.
This is made even more frustrating by the fact that I have a PC at home, and I’m not exactly sure how to modify the voice recognition software. I’ll do research, of course, but it would help if you knew anything about how to do the same on a PC, instead of a Mac…
Sorry if I’m sounding monumentally ignorant, but I’m relatively new to this.
You can ignore that last post I made. I figured out how to unblock everything, and I got your “Thank you” script. How exactly would I program this to the speakable commands list?
Hi Justin,
What scripting languages are you using to integrate all this? I am interested because I am working on a similar project but I am starting from the automation side rather than the computer scripting side. Been playing around with a program called Homeseer that allows you to control your home appliances and light, etc. through your computer. It can announce information and schedule events. The only problem is that now I want to be able to accurate talk to the computer and give it personality.
Any information you have about your scripting endeavors would be much appreciated as my scripting knowledge is limited.
Thanks,
Mike
-Also Building Jarvis.
Hi everybody as from above i was watchin iron man for like the 5000th time and had a thought how good would it be to have a jarvis type A.I for my home so i started doing some research,, and its actuly not that hard to do all you need is knowledge of Microsoft Visual Basic, but making they system able to talk back to you is a complete differant thing all together, but after lookin into it Mircosoft Windows 7 come preinstalled with a Speech Reconition Software were u can control your system with your voice there has been a number of tests on this and seems to work quiet well, i used it and withing the first 10 minutes i was searching the internet with my voice and using my programs, hell im evening writen this talking to my computer, yes there is a problem with the Windows 7 speech reconition software due to human error with your accent there is more software been developed to fit with the program to help assist people with the problems with accents, might be a good idea looking into all of this.
Hope this helps guys nd girls
Hi !
I TOO CREATED JARVIS HAVE ACHIEVED A LEVEL WHERE I ASK HIM TO TELL ME THE MEANING OF ANY WORD AVAILABLE IN THE DICTIONARY SAY
” JARVIS CAN I HAVE A LOOK AT THE MEANING OF SERENE”
OR
“JARVIS LOOK OR SERENE IN THE DICTIONARY”
OR
“JARVIS CAN I HAVE SERENE FROM THE DICTIONARY”
THE CODE SNIPPET READS THE KEYWORDS “I’, “HAVE” , “MEANING” , “SERENE” , “DICTIONARY”
IT MANIPULATES TO LOOK FOR THE MEANING OF SERENE AND LOOKS FOR THE MEANING IN THE DICTIONARY IN ACCESS AND SAYS
“SIR, SERENE ACCORDING TO ME MEANS PURE”
MAY BE REPLACED BY THE ACTUAL MEANING IN THE DICTIONARY DATABASE’
IT THEN SAYS
“ANY MORE ORDERS FOR ME SIR ?”
AND THEN STANDS BY
OK IF YOU WANT TO CONTACT ME OR COMMENT ON THIS POST THEN DROP ME A LINE ON NOREPLY.DHEERAJ@GMAIL.COM AS I MAY NOT BE ABLE TO POST OR REPLY ON THIS SITE
BYE !
Now, being completely new to scripting, I find this task is probably a little more then I can handle, but here goes.
Justin, using your template for the open browser, I attempted to get my own Jarvis to not only open my safari but go to specific pages upon opening the browser. Here’s the basic script so far…
tell application “SpeechRecognitionServer”
tell application “Safari” to activate
set theResponse to listen for {“Facebook”,”Post Secret”} with prompt “Where would you like to go Sir?”
if the Response is “Facebook” then
tell application “Open Facebook” to activate
say “Opening Facebook.”
else
if the Response is “Post Secret” then
tell application “Open Post Secret” to activate
say “Post Secret.”
end if
end tell
Although it may look vastly incomplete essentially it should work (to my limited knowledge). The applications “Open Post Secret” and “Open Facebook” were created with Automater and run perfectly by themselves, I run into an issue as soon as I try to save the program as it tells me that it expects to find end instead of end tell. Where in the script have I gone wrong?
Sorry for the double post, but I figured out the issue. I now grasp that I can’t have two set responses under the template, only a response and an alternative. Would there be a way to script this for more then two options?
Hey, thought this was really good! Just wondering if you’re still working on this at all?
Tony Stark’s A.I Butler is absolutely remarkable, however many people do not realize that this type of A.I is possible (Its from a fantasy movie – how the hell can it be real right? wrong) which is why I have decided to make one of my own and to be honest the hardest part was selecting a name (I went with Alfred) (Yes, Batman’s butler…)
Hey check out http://www.jabberwacky.com. It might help you solve some issues for the personality aspect.
how to force SR to not complete a command until it hears a certain word in windows 7
pls reply on mail or this site ASAP..pls help me guys
Hey there!
It warns me if there are any recent things I need to know, or write something for me, or tells people I’m busy etc. The list I could be able to manage to control is:
We have been working on the same thing for almost a year, and we don’t know each other. I have adapted the same system into aMSN too
Mail, Address Book, iTunes, Safari, iChat, aMSN…
I have been working on the implementation of ALICE in this system. I believe we can create this together. I can’t see any obstacles now
I felt superb after seeing this blog! Please contact me asap! If we can manage to integrate ALICE into what we both have done so far, this thing would literally become the Jarvis.
I also tried to use this wirelessly. Main problem is the microphone. It ONLY supports bluetooth headsets, but the quality sucks so bad, that even I couldn’t understand what I say, not to mention what “Jarvis” would.
I’m a Music Technologies Undergrad. Student. I believe I can work out the audio problem soon. I mean, there must be a good efficient way.
Please please please contact me.
I can also use help from people who can use Java and/or AppleScript.
I hope you don’t have a spambot too
here is my mail : c h e a t e r . b o s s [ a t ] g m a i l . c o m
Future needs us
Cheers
Also, physicistjedi : I have seen yours too !!!! It was one of my motivations
Mike, contact me maybe we could help each other too…
Well actually we need to be organized. There are at least 4 persons here trying the same. We can do this together? Why wait desperately? How about a sourceforge project ?
Hey everyone,
Sorry for a late reply. First off, I want to thank everyone for the support and help. Keep it up.
I know its been awhile. Life and other projects have gotten in the way but I made a break through. My research found an old supported AIML node called <system>. This node is still recognized by ProgramD, but is not recognized by the current Pandorabots project. This <system> tag is my new link.
The <system> tag tells the bot to execute an OS related function. ProgramD runs through Terminal, and since Terminal allows one to execute Applescript, I input that specific terminal code within the response tag. For instance, this code will execute an Applescript I made that checks email:
<pattern>
<template>CHECK MY MAIL</template>
<response>
<system>osascript checkmail.script</system>
</response>
</pattern>
So if I type “Check my email” into iChat, Bixby will execute the applescript. This means I can forgo using SR and use iChat exclusively. However, this means that I must type my commands and cannot speak to execute them. I am confident that I can find a solution. What do you guys think?